{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handwriting Recognition - USPS Post Dataset\n",
    "\n",
    "## Problem:\n",
    "\n",
    "This project introduces us to a major topic in Supervised Machine Learning, which is __Classification__. We shall be studying the multiclass classification problem, by applying supervised classification algorithms on two datasets provided:\n",
    "\n",
    "    1. MNIST Dataset\n",
    "\n",
    "    2. USPS Dataset\n",
    "    \n",
    "In particular, we shall be applying the following algorithms for the task of __Handwritten Character Recognition__:\n",
    "        1. Logistic Regression\n",
    "    \n",
    "        2. Single layered Neural Networks\n",
    "    \n",
    "        3. Convolutional Neural Networks\n",
    "    \n",
    "This is one of the standard problem in Machine Learning field, and many new algorithms are often compared by their output on this task.\n",
    "\n",
    "\n",
    "While solving the problem of Handwriting Recognition, we shall encounter the __NO FREE LUNCH THEOREM__. Our objective is to verify that this theorem holds even in this case, as we shall see in subsequent sections.\n",
    "\n",
    "\n",
    "## Dataset:\n",
    "\n",
    "    1. We have been provided with the MNIST handwritten digits dataset, \n",
    "    which consists of 70000 grayscale images, representing 10 digits - \n",
    "    0 to 9. The images are each of 28 x 28 pixel resolution.\n",
    "\n",
    "    2. We shall be using the MNIST dataset for training our machine \n",
    "    learning models.\n",
    "    \n",
    "    3. The other dataset is the USPS Dataset. It consists of approximately \n",
    "    20000 pixels, which are at resolution of 100ppi. We need to preprocess \n",
    "    these images to have same resolution as our MNIST data images, so that \n",
    "    our trained models can be applied on this dataset.\n",
    "    \n",
    "## Solution:\n",
    "\n",
    "First we begin by training 3 Machine learning algorithms on the MNIST dataset. Then we shall use these trained models for predicting labels of images in USPS Dataset.\n",
    "\n",
    "Let us begin by first importing the MNIST Dataset, and preprocessing them in required output format for training our algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import cPickle as pickle\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us begin by importing the MNIST dataset, which is stored in a pickle file. We shall import each image which is 28 x 28 pixel matrices, and we shall flatten them out to a 1 x 784 dimension vector.\n",
    "\n",
    "### Split the MNIST Dataset\n",
    "\n",
    "We shall also split the dataset into __Train, validation and test data__. We shall have the following split for the data:\n",
    "    1. Train set -  50000 images\n",
    "    2. Validation set - 10000 images\n",
    "    3. Test set - 10000 images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pickle_file = 'mnist.pkl'\n",
    "image_size = 28\n",
    "num_labels = 10\n",
    "\n",
    "def reformat(labels):\n",
    "    # Map 0 to [1.0, 0.0, 0.0 ...], 1 to [0.0, 1.0, 0.0 ...]\n",
    "    labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)\n",
    "    return labels\n",
    "\n",
    "\n",
    "with open(pickle_file, 'rb') as f:\n",
    "    save = pickle.load(f)\n",
    "    train_dataset = save[0][0]\n",
    "    train_labels = reformat(save[0][1])\n",
    "    raw_train_labels = save[0][1]\n",
    "    valid_dataset = save[1][0]\n",
    "    valid_labels = reformat(save[1][1])\n",
    "    raw_valid_labels = save[1][1]\n",
    "    test_dataset = save[2][0]\n",
    "    test_labels = reformat(save[2][1])\n",
    "    raw_test_labels = save[2][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Training set', (50000, 784), (50000, 10))\n",
      "('Validation set', (10000, 784), (10000, 10))\n",
      "('Test set', (10000, 784), (10000, 10))\n"
     ]
    }
   ],
   "source": [
    "print('Training set', train_dataset.shape, train_labels.shape)\n",
    "print('Validation set', valid_dataset.shape, valid_labels.shape)\n",
    "print('Test set', test_dataset.shape, test_labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression\n",
    "\n",
    "Let us train our first machine learning model, which is the multiclass logistic regression. The model parameters are as follows:\n",
    "\n",
    "    1. Input: 784 element vector, representing a flattened image.\n",
    "    \n",
    "    2. We add a 1 to the vector, to handle the bias term. \n",
    "    Hence our input vector consists of 785 elements.\n",
    "    \n",
    "    3. The output is a 10 dimensional 'one-hot encoded' vector, \n",
    "    representing a single digit output.\n",
    "    \n",
    "    4. We shall be using batch gradient descent, to speed up \n",
    "    the training process,and converge the algorithm faster.\n",
    "    \n",
    "    5. We shall be using the Cross Entropy Cost function for \n",
    "    optimising our model weights.\n",
    "    \n",
    "    6. Since, the problem is actually a classification problem,\n",
    "    a Softmax layer is added after the regression output, to convert \n",
    "    the output values into class probabilities.\n",
    "    \n",
    "The logistic classification model can be represented as:\n",
    "\n",
    "    a[i] = Softmax(W*x[i] + b)\n",
    "\n",
    "    y[i] = exp(a[i])/sum(a[k])\n",
    "    \n",
    "    where x[i] is ith training sample and y[i] is the \n",
    "    predicted output vector corresponding of ith example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def accuracy(y, t):\n",
    "    count = 0\n",
    "    for i in range(len(y)):\n",
    "        if y[i] == t[i]:\n",
    "            count += 1\n",
    "    return float(count)/len(y)\n",
    "\n",
    "def one_hot_encoding(t):\n",
    "    return np.argmax(t, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_data(y_values1, label, axis_dim, xlabel, ylabel, title):\n",
    "    plt.plot(y_values1, 'b-',label=label)\n",
    "    plt.axis(axis_dim)\n",
    "    plt.ylabel(ylabel)\n",
    "    plt.xlabel(xlabel)\n",
    "    plt.title(title)\n",
    "    l = plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def add_ones(X):\n",
    "    return np.hstack((np.zeros(shape=(X.shape[0],1), dtype='float') + 1, X))\n",
    "\n",
    "def calculate_entropy_loss(X, w, y):\n",
    "    loss = 0\n",
    "    t = np.dot(X, w)\n",
    "    for i in range(len(t)):\n",
    "        loss += -1 * np.dot(y[i], np.log(t[i].T))\n",
    "    return float(loss)/len(t)\n",
    "\n",
    "def softmax(t):\n",
    "    prob_matrix = []\n",
    "    for i in range(len(t)):\n",
    "        prob_vector = []\n",
    "        sum_exp = 0\n",
    "        for j in t[i]:\n",
    "            sum_exp += np.exp(j)\n",
    "        for k in t[i]:\n",
    "            prob_vector.append(float(np.exp(k))/sum_exp)\n",
    "        prob_matrix.append(prob_vector)\n",
    "    return np.array(prob_matrix)\n",
    "            \n",
    "def get_probabilities(X, theta):\n",
    "    t = np.dot(X, theta)\n",
    "    prob_vector = softmax(t)\n",
    "    return prob_vector\n",
    "\n",
    "def train_log_regression(X, y):\n",
    "    X = add_ones(X) # Bias term\n",
    "    maxiter = 100\n",
    "    batch_size = 20\n",
    "    n = X.shape[1] \n",
    "    m = X.shape[0]\n",
    "    theta = np.random.rand(n, len(y[0])) # Initialise random weights\n",
    "    lmbda = 0.01\n",
    "    alpha = 0.01\n",
    "    max_error = 0.1\n",
    "    loss = 10\n",
    "    err_iteration = []\n",
    "    train_accuracy = []\n",
    "    validation_accuracy = []\n",
    "    print \"num_iter | Loss\"\n",
    "    for iteration in xrange(maxiter):\n",
    "        start = 0\n",
    "        for i in range(m/batch_size):\n",
    "            out_probs = get_probabilities(\n",
    "                X[start:start+batch_size], theta)\n",
    "            grad = (1.0/batch_size) * np.dot(\n",
    "                X[start:start+batch_size].T,\n",
    "                (out_probs - y[start:start+batch_size]))\n",
    "            g0 = grad[0]\n",
    "            grad += ((lmbda * theta) / batch_size)\n",
    "            grad[0] = g0 \n",
    "            theta -= alpha * grad\n",
    "\n",
    "            # calculate the magnitude of the gradient and check for convergence\n",
    "            loss = calculate_entropy_loss(\n",
    "                X[start:start+batch_size], theta, y)\n",
    "            start += batch_size\n",
    "        err_iteration.append(loss)\n",
    "        pred_output_train = np.dot(X, theta)\n",
    "        train_accuracy.append(\n",
    "            accuracy(raw_train_labels,\n",
    "                     one_hot_encoding(pred_output_train)))\n",
    "        pred_output_valid = np.dot(add_ones(valid_dataset), theta)\n",
    "        validation_accuracy.append(accuracy(\n",
    "                raw_valid_labels, one_hot_encoding(pred_output_valid)))\n",
    "        if np.abs(loss)< max_error or math.isnan(loss):\n",
    "            break\n",
    "    return theta, err_iteration, train_accuracy, validation_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_iter | Loss\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/site-packages/ipykernel/__main__.py:8: RuntimeWarning: invalid value encountered in log\n"
     ]
    }
   ],
   "source": [
    "weights, err_iteration, train_accuracy, \n",
    "    validation_accuracy = train_log_regression(train_dataset, train_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have trained our logistic regression model, let us view how the model training has improved over iterations of our training.\n",
    "\n",
    "### Decreasing Loss as Model Trains "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEZCAYAAAC99aPhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNW5x/Hvy7AICKKgqCCbGFEi7qhBA6KiRnG5KkEi\nCkaSIBKXe4l7IF4SNcYtKsSoQUMwxIgLKu4wEbcLooKAC6KAAgKiCG4sw3v/ODXSjDNDDzXd1cvv\n8zz90F1dXeetsu13zjl1zjF3R0REZGvVSToAERHJb0okIiISixKJiIjEokQiIiKxKJGIiEgsSiQi\nIhKLEonkHTP70Mx6Jh1HrjKz0WZ2ZcIxzDazHycZg2SPEolUSz/a+cfdB7v77wHMrLuZfZTJ8sxs\njJldUyGGH7r7C5ksV3KHEolIHjOzki3tAmz1qOM0ji+iRCJbz8wGmdk8M/vUzB4xs11S3rvZzJaZ\n2RdmNtPM9o62/8TM5pjZajP7yMwu2cLx50b7zjaz/VLe3j867udm9k8zqx99ppmZPWZmy81sZfS8\nVcoxp5jZNWb2YnTcp8xsh5T3zzazBWa2wsyuSq2RWXCZmb0fvT/ezJrV4PrsHG0fZWY3VNj3ETO7\nKHq+i5k9GJ3DfDMbmrLfcDP7t5mNNbNVwDmVlDsmOsdGwCRgVzNbE53vztWdh5m1NbONZnaumS0E\nno+2P2BmS6PrXWpme5WfI/Az4DfR8R+Ntqdet/pmdouZLTazj6PvRr3ove7l34Po+7LYzAaknEva\n3xdJkLvroUeVD+BDoGcl23sCK4B9gXrAn4H/RO/1AqYDTaLXewIto+dLgB9Fz7cD9qui3DOAj4AD\notcdgN1SYnoVaAk0A+YCv4je2wE4FWgANAb+BTycctwpwDxg92ifKcAfovf2BtYAhwF1gRuAteXn\nD1wIvAzsEp3zaOD+KuKv7vocASxM2bcZ8HV0Pga8BlwJlADtgPeBY6J9h0cx9Y5eN6ik7DHANdHz\n7sCiCu9XeR5AW2AjcC/QsPz4wACgUbT/TcAblZVX2fcGuCYqr3n0eAn4XUp866PzKgGOB74CtqvJ\n90WPhH8nkg5Aj9x+UHUiuRu4LuV14+gHrg1wJPAOcAhgFT63ABhElGSqKfcpYGg1MZ2Z8vp6YFQV\n++4HrEx5PQW4IuX1YGBS9PxqYFzKew3ZPJHMBY5MeX8XYB1QJ83rsw5ok3IdDo+enwc8Fz0/BFhQ\n4ViXAfdEz4cDpVu4dltKJFWeR5RIyoC21Ry/GSHZNKlYXmXfG0IiPDblvV7ABynxfZV6DYFlQNea\nfF/0SPahpi3ZWrsCC8tfuPtXwGdAK3efAtwO3AEsM7O/mNm20a6nAScAC6NmpkOrOP5uwPxqyl+W\n8vxrYFsAM2toZndGzVOrgP8AzczMUvb/pLLPRuf0Xce0u38DrEzZty3wsJl9ZmafEX6Q1xNqEhVV\ndn1WAuXNbP8Czoye9wPGRc/bAK3KyzCzz4HLgZ1Sjh238zyd8/i4/ImZ1TGz66KmsFWEJOFAizTL\n2xVYlPJ6YbSt3Ep335jyOvW/SbrfF0mQEolsrSWEHyQAzKwxodliMYC73+7uBxGai/YEhkXbZ7j7\nKcCOwKPAA1Uc/yNC81NN/Q+wB3CwuzcDym9Btao/8p2lQOvyF2bWkHBO5RYBx7v7DtFje3dv7O5L\nKzlWtdcH+Cdwupm1IdRCJkTbPyL8tZ5axnbu3jvl2DXpPK9s33TOI/Vz/YDehBpGM0Jzm7Hpmm4p\nns2uRfR8SVrBp/99kQQpkUg66ptZg5RHCeGHcKCZdTGzBsAfgFfcfZGZHWRmXc2sLvAN8C2w0czq\nmVk/M2vq7mWE/oiyKsq8G/gfMzsAwMx2N7Pd0oh126jM1VEn+oganOeDQG8zOzTqDK742TuBP0Q/\n/pjZjmZ2UhXHquz6vOruiwDc/U1CDeVu4Cl3Xx19bhqwxsx+Y2bbmFmJmXU2s4NqcB6plgHNzaxp\nDc6jYtJtQmji+zxKiNeyefJYRujDqso/gavMrIWZtSA0IY7dUuA1/L5IgpRIJB1PEJobvon+He7u\nzxN+EB4i/JXdnk1NNU2BuwhNXR8CnxI6rgH6Ax9GTSS/IPy1+z3u/iDwe+B+M1sNPEzoSIfq/wK+\nhdAp/Cmhg3dSxUNX9UF3nwsMJTQ7LQFWA8sJP6IAtxL+Kn7GzL6Ijt+1imNVdn36VtjtfuAoNjVr\nETXxnEjo2/kwKv8uwjVN13fn6O7vEn7IP4iasnZO4zwqXqO/E2oxi4HZ0f6p7gE6R8d/qJJjjCTc\nQDALmBk9/3068ZPm90WSZe7JLmxlZv9N+JFp4e6fVfL+cYQfhzqEDsfrsxyiFKnor+9VQEd3X7il\n/UWKVaI1EjNrDRxDSqdkhffrEDptjwU6A2eaWafsRSjFxsxOjDrsGwM3ArOURESql3TT1s1EnbBV\n6ArMc/eF7r4eGA+cnJXIpFidTGjW+pjQ2V+xOUpEKqibVMFR595H7v7W5ndmbqYVm9/q+DFVtEmL\n1AZ3H0QYtyAiacpoIjGzZ9n83vTyeX+uAq4gNGulviciInkmo4nE3Y+pbLuZ/ZBwL/rMaKBYa2CG\nmXV19+Upuy4mDNAq15pN9+FXdtxk7xwQEclD7h7rD/lE+kjcfba77+zuHdy9PaHJav8KSQTCfE0d\no4nk6hPaqydu4dh6uDN8+PDEY8iFh66DroWuRfWP2pB0Z3s5J2rasjDz6eMAHgYhXQA8A8wBxrv7\n24lFKSIi35NYZ3sqd++Q8nwpYUBW+eunCFNsiIhIDsqVGonUsh49eiQdQk7QddhE12ITXYvalfjI\n9tpkZl5I5yMikmlmhsfsbM+Jpi0RkUxp164dCxdqcoK2bduyYMGCjBxbNRIRKWjRX9xJh5G4qq5D\nbdRI1EciIiKxKJGIiEgsBZdIxo2Dr75KOgoRkeJRkImkVSs491x49VVQ06iI5Kr27dszefLkpMOI\nreASyaRJ8O670KkT9O8P++4Lt98OX3yRdGQiIoWp4BIJQMuW8JvfhIRyyy3wwgvQrh0MGgQzZiQd\nnYhIYSnIRFKuTh3o2RMeeADefhvat4fTToOuXWHMGPj666QjFBGBdevWcdFFF9GqVStat27NxRdf\nzPr16wFYuXIlvXv3Zvvtt6d58+Z07979u89df/31tG7dmqZNm7LXXnsxZcqUROIv6ESSaued4Yor\nYP58+O1v4cEHoU0buOQSeO+9pKMTkWI2cuRIpk2bxqxZs5g5cybTpk1j5MiRANx4443sttturFy5\nkuXLl/OHP/wBgPfee4877riDGTNmsHr1ap5++mnatWuXSPxFk0jKlZTAiSfCE0/A9OnQoAEccQQc\nfTQ89BBs2JB0hCKSTWa184jj/vvvZ/jw4TRv3pzmzZszfPhwxo4dC0C9evVYunQpH374ISUlJXTr\n1g2AkpIS1q1bx+zZs9mwYQNt2rShffv2cS/HVim6RJKqfXu49lpYtAh+/nO4+WZo2xZGjIDFVS6f\nJSKFxL12HlujfLT5kiVLaNNm0xp+bdu2ZcmSJQAMGzaM3XffnV69etGxY0euv/56AHbffXduueUW\nRowYQcuWLenXrx9Lly6NfT22RlEnknINGsCZZ8LUqfDUU7BiBeyzT+hPee452Lgx6QhFpFCZGa1a\ntdpsPrCFCxey6667ArDtttvypz/9ifnz5zNx4kRuuumm7/pC+vbty9SpU7/77GWXXZb9E0CJ5Hv2\n2QfuuAMWLoRjjgl9KJ06hdrK558nHZ2IFJLyua/69u3LyJEj+fTTT/n000/53//9X/r37w/AE088\nwfz58wFo0qQJdevWpU6dOrz33ntMmTKFdevWUb9+fRo2bEidOsn8pCuRVKFJE/jVr2DmzHCH14wZ\noSls4MDQtyIiEpdFnStXX301Bx54IF26dGHffffloIMO4sorrwRg3rx5HH300TRp0oRu3boxZMgQ\nunfvztq1a7nsssvYcccd2XXXXVmxYgXXXnttMudRSLNiZnr23+XL4W9/gzvvhObN4fzzoW9faNQo\nY0WKSEya/TfI5Oy/SiRboawMnn4aRo0K07D07x9qL3tqQWCRnKNEEmga+RxTUgI/+Qk8/ji89hps\nsw38+MfhFuIJEyAaRyQiUhRUI6kla9eGcSijR4dBj4MGhUerVomEIyIR1UgC1UjyQPktxC+8EJq9\nKt5CrO+xiBQq1UgyaM2aMK396NHw7behH2XAANh++6QjEykeqpEE6mxPU64lknLu8PLLIaE88QSc\nemq44+ugg5KOTKTwKZEEmUwkdeN8WNJjBt26hUf5LcRnnAEtWsDgwbqFWCST2rZt+914jWLWtm3b\njB1bNZKElJWF6VhGj950C/HgwfCDHyQdmYgUE3W257GSEjjhhM1vIdYsxCKSj1QjySFr14ZxKKNH\nw4cfbrqFOJq7TUSk1qlGUmAaNIB+/cIsxJMmwSefQOfOcPrpMHmybiEWkdykGkmOW70a/vGPMB3L\nhg2hH+Wcc6BZs6QjE5FCoNt/KyjERFLOPdRURo8OnfSnnRZuIT7ggKQjE5F8pkRSQSEnklTLlsHd\nd8Nf/wq77BJqKX36QMOGSUcmIvlGiaSCYkkk5crKQl/KqFHhzq8BA8Lo+d13TzoyEckX6mwvciUl\n0Ls3PPlkGItiBoceCscdBxMnhkQjIpJpqpEUmG+/hQceCH0pS5bAL38JP/85tGyZdGQikotUI5Hv\n2WYbOPtseOUVeOSRMB6lU6cwM/HUqbqFWERqn2okRWDVKvj730NfSr16oXP+rLOgadOkIxORpKmz\nvQIlkuq5w5QpIaE8/3yopQweHNZNEZHipKYtqREz6NkTHnwQZs+GHXeEY48NywT/85+wbl3SEYpI\nPlKNpMitXx/u8LrjDpg7N3TM//KX0KZN0pGJSDaoRiKx1asXRslPnhyavb78EvbfH04+OSwZvHFj\n0hGKSK5TjUS+56uv4P77Q1/KmjVhkOPAgdC8edKRiUhtU41EMqJx4zB9/euvhwkjZ84Mo+UHDIBp\n05KOTkRyjWokkpYVK2DMGPjLX2CHHcKEkVoiWCT/6fbfCpRIMq+sLPSdjBoVpmU5++xwC/EeeyQd\nmYhsDTVtSdaVlMBPfhKWCJ4+HerXh8MPD7cRP/KIlggWKUaqkUhsa9eGsSl33AEffxxuHz7vPM3v\nJZIP8rpGYmbXmNlMM3vTzJ4zs9ZV7Lcg2u8NM1NXbw5q0AB+9jN4+eUwJmXRIs3vJVJMEquRmNm2\n7v5l9HwosK+7n1fJfh8AB7r752kcUzWSHLFqFYwdG/pS6tYNnfNnnQVNmiQdmYikyusaSXkSiTQG\nPq1iV0N9OXmnWTMYOjSMlv/zn8PcXm3bwpAhYXoWESkcif5Am9lIM1sEDACurWI3B541s+lmNihr\nwUmtMIMjjwx9KG+9Feb36tULuneHf/1L83uJFIKMNm2Z2bNAaperERLDle7+WMp+lwKd3H1gJcfY\nxd2XmtmOwLPABe7+YhXl+fDhw7973aNHD3r06FEr5yK1Z/36cIfXqFHwzjth8OMvfgGtK+0lE5Ha\nVFpaSmlp6Xevf/e73xXGOBIz2w2Y5O7VTmhuZsOBNe5+UxXvq48kz8ydG1ZzHDcOevQIfSlHHRVq\nMiKSeXndR2JmHVNengK8Wck+jcxs2+h5Y6AXoBb2ArL33nDbbbBwYWjyuvjicMfXrbeGDnsRyX1J\n3rX1IPADoAz4ABjs7svNbBfgLnc/0czaAw8TmsPqAuPc/bpqjqkaSZ5zh5deCs1eTz4Jp58eOuj3\n2y/pyEQKk6ZIqUCJpLAsWwZ33w133hn6T84/H844I4xbEZHaoURSgRJJYdqwIUzJMno0vPkmnHtu\nGD3frl3SkYnkv7zuIxFJV926cMopYbLIqVPDlCwHHQS9e8NTT2nxLZGkqUYieenrr8M683fcAatX\nh8W3zj03THEvIulTjUSKVqNGYX35GTPC4luzZkGHDmElx9deSzo6keKiGokUjBUr4J57wuJbO+0U\n7vbq0wcaNkw6MpHcpc72CpRIBMLiW08+GW4hnj491FIGD4b27ZOOTCT3qGlLpBIlJXDiiTBpUljF\nceNGOPhgOOGEsK2sLOkIRQpLjWokZrY9sJu7z8pcSFtPNRKpyjffwPjxoXP+s882dc63aJF0ZCLJ\nykqNxMxKzaypme0AvA7cZWaVznUlkqsaNtzUET9+PMyZAx07woABoflLRLZeOk1b27n7auC/gL+7\n+yHA0ZkNSyRzunaF++6D998Pc3316RO23XtvqLmISM2kk0jqRvNf9QEez3A8IlnTogX85jchofz2\nt/DAA9CmTdj2wQdJRyeSP9JJJNcATwPvu/t0M+sAzMtsWCLZU1nn/CGHbOqc18h5kerp9l+RSlTs\nnB88OHTON2+edGQitStbne1/jDrb65nZ82a2wszOilOoSK4r75yfPj1MxTJ7duic18h5ke9Lp2mr\nV9TZfiKwAOgIDMtkUCK5wiw0c913H8ybFxbdOv30Tdu+/TbpCEWSl1Zne/TvCcC/3f2LDMYjkrNa\ntIBLL4X58+Gqq0JNpU2bsG3BgqSjE0lOOonkcTN7BzgQeN7MdgT0d5gUrZKSTVPYv/QSrF+vae2l\nuKXV2R4NRvzC3cvMrBHQ1N0/yXh0NaTOdknK11/D/feHzvkvvwyrOQ4YANtvn3RkItXLyqSNZlYP\nGAz8ONr0H+Av7r4+TsGZoEQiSXOHV14JCWXSJK05L7kvW4nkbqAecF+0qT9Q5u7nxSk4E5RIJJeU\nrzn/l7+EvpQhQ0JiqV8/6chENslWIpnp7vtuaVsuUCKRXLRhAzz2WKilzJ4NgwaFNedbt046MpHs\nTSNfZma7pxTaAdBE3CJpqlsXTj0VnnsOpkyBVaugSxc47TSYPDk0h4nks3RqJEcBY4APAAPaAgPd\nfUrmw6sZ1UgkX6xZA2PHhloKhM75s8+GJk2SjUuKT9ZWSDSzBsCe0ct33X1tnEIzRYlE8o07lJaG\nhDJ5MvTrF/pS9tor6cikWGQ0kZjZf1X3QXd/KE7BmaBEIvns44/hr38Nj86d4YILwtiUunW3/FmR\nrZXpRDKmms+5u58bp+BMUCKRQrB2LUyYALffHpLLr34F550HO+2UdGRSiLLWtJUvlEik0LzxRmj2\nmjAhTHU/ZEiY58ti/W8vsokSSQVKJFKoPvsMxoyBUaPCaPkLLoCf/jTMUiwShxJJBUokUug2boQn\nnwy1lNdeC9PaDx4M7dolHZnkq2yNIxGRHFGnzqaVG19+OQx2PPBAOPlkePZZTRgpyUhnHMkM4G/A\n/e7+eVai2kqqkUgx+uqrMGHk7beHjvohQ+Ccc6Bp06Qjk3yQrRrJT4FdgelmNt7MjjVTV59Irmjc\nOEy78uabcNdd8OKLoalryBCYOzfp6KQYpN1HYmZ1CKskjiZMkTIGuNXdP8tceDWjGolIsHgx3Hln\nSCx77w1Dh4a7vjQmRSrK5sj2LsBA4CfA08A44HCgv7vnzATZSiQim1u3bvMxKYMHhzEpLVokHZnk\nimzN/jsDWAXcA0xInR7FzB5y92pHwGeTEolI1d54IySUhx4KnfMXXBBWdpTilq1E0sHdP4hTSLYo\nkYhs2cqVcM89YUzKzjuHhHLGGdCgQdKRSRKylUiaA8MJTVkOvAhc4+4r4xScCUokIukrK4MnnoDb\nboO33god9r/6FbRqlXRkkk3ZumtrPLACOA04PXr+rziFikjySkrgpJPC+JPS0rBOyj77QJ8+MHWq\n1kmR9KVTI5nt7j+ssO0td98no5FtBdVIROJZvRruuy+MnG/QIDR7/exn0KhR0pFJpmSrRvKMmfU1\nszrRow/hzi0RKTBNm4ZbhefOhRtugIkTw3rzw4bBhx8mHZ3kqnRqJGuAxkD55At1gK+i5+7uOTN+\nVjUSkdr3wQehY/7ee+FHPwqJ5uijNQNxodCkjRUokYhkTvlULLfdBuvXh2YvLQ+c/7I5IPEk4MfR\ny1J3fzxOoZmiRCKSee7wwgshoUyZAmedFZLKHnskHZlsjaz0kZjZdcCFwNzocaGZXRunUBHJX2bQ\nvTs8+GCY36txY+jWDY4/PkxxrxmIi086fSSzgP3cfWP0ugR4w927ZCG+GlGNRCQZ334L48fDn/8M\na9aECSMHDoTttks6MtmSbK5H0izlub4aIrKZbbaBAQNgxozQKf/qq9C+fUgob7+ddHSSaekkkmuB\nN8zsXjO7D5gB/D6zYYlIPjILzVzjx4fR8jvsAD16QK9e8PjjYTS9FJ5qm7aidUdaAxuAg6PN09z9\nk9gFm10DnEyYduVTYIC7f1zJfscBtxCS3j3ufn01x1TTlkiOWbsWHngAbr0VPv88dMwPHAjNmm35\ns5J52ZprKyOj2M1sW3f/Mno+FNjX3c+rsE8d4D3gKGAJMB3o6+7vVHFMJRKRHOUemrxuuy10yvfr\nF5LKXnslHVlxy1YfyetmdvCWd6uZ8iQSaUyolVTUFZjn7gvdfT1h3q+TazsWEck8MzjssDAWZc4c\naN4cjjwSjj02TB6pu73yVzo1kneAjsBCwoh2I4xoj33XlpmNBM4GvgYOcfcvKrx/GnCsu/8ien0W\n0NXdf13F8VQjEckjqc1eq1ZtavbS3V7ZUxs1knQW3jx2aw9uZs8CLVM3EfpErnT3x9z9KuAqM7uU\n0A8ycGvLKjdixIjvnvfo0YMePXrEPaSIZEiDBtC/fxjU+MorodnrmmtCs9fQobDnnklHWHhKS0sp\nLS2t1WOmUyMZ6+79t7QtVhBmuwGTKvbFmNmhwAh3Py56fRmhNlRph7tqJCL5b/FiGD06rDd/wAHw\n61+H5q866Q5WkBrJVh9J5wqFlgAHxik0Ok7HlJenAG9Wstt0oKOZtTWz+kBfYGLcskUkd7VqBSNH\nwsKF8NOfwhVXhA75228Pgx0l91SZSMzs8mjm3y5mtjp6rAGWA4/WQtnXmdksM3sD6AH8d1TuLmb2\nOIC7lwEXAM8Ac4Dx7q7hTSJFoHyQ4+uvw913h8W32rWDiy+G+fMTDk42k07T1rXufnmW4olFTVsi\nhW3RorDo1t/+Fu4Au/BC6NlTU9rHkc3Zf1sBbUnpnHf3F+IUnAlKJCLF4euvYdy4cLeXWehH0UqO\nWydbAxKvI/RNzAXKJzhwdz8pTsGZoEQiUlzcYfLkkFBeeQXOOy/M79W6ddKR5Y9sJZJ3gS7uvjZO\nQdmgRCJSvN5/P9w+PHYsHHMMXHQRHHqomr22JFt3bX0A1ItTiIhIpnXsGGomCxaEJYH794dDDgkj\n6detSzq6wpZOjWQCsC/wPPBdraSq0eVJUo1ERMqVlcGkSXDLLfDOO3D++fDLX0KLFklHlluy1bR1\nTmXb3f2+OAVnghKJiFRm1qyw6NaECXD66eFurx/+MOmockM279pqCLRx93fjFJZpSiQiUp0VK+DO\nO2HUKOjcOfSjHH98cY+az1aNpDfwJ6C+u7c3s/2Aa3TXlojkq3XrwmSRN98cRsv/+tdh8OO22yYd\nWfZlq7N9BGE691UA7v4m0CFOoSIiSapfP0wU+dprYXDjlClh1PywYWFqFqmZdBLJ+orTuwNaOUBE\n8p4ZHH546DuZPj100O+/P/TpE8alSHrSSSRzzKwfUGJme5jZbcDLGY5LRCSr2reHm24Ktw936xZG\nyh96KPzrX7BhQ9LR5bZ0+kgaAVcCvaJNTwMj3f3bDMdWY+ojEZHaUlYGEyeGfpQFC8L6KIMGFd5a\n81m7aytfKJGISCbMmBESyqRJoaZy4YVhAGQhyFZnu4hIUTvwQPjHP+Ctt8KdXYcdBqeeCi+8EOb7\nKnaqkYiI1NBXX8Hf/x5qKU2bwiWXwBlnQL08nExKTVsVKJGISDZt3AhPPBE66d9/P4xHybd+lKw0\nbZnZH82sqZnVM7PnzWyFmZ0Vp1ARkUJQpw707h3GoTz6KMycCR06hD6UDz9MOrrsSaePpJe7rwZO\nBBYAHYFhmQxKRCTfHHBA6EeZNSssE3zwwaG569VXk44s89JJJOWrIp4A/LuSwYkiIhJp3Rquvz7c\nMnzEEdCvX5jWfsKEcEtxIUp3hcRTgG8IU6U0Ax5390MyH17NqI9ERHLNhg3wyCNw442wfDlcfDEM\nHAiNGycdWZDN2X93AL5w97JogGJTd/8kTsGZoEQiIrnspZdCQpk6NayNcsEFsPPOycaUrc72Mwjz\nbZWZ2VXAP4Bd4xQqIlKMunWDhx6Cl1+Gzz6DvfYK68zPnZt0ZPGk00dytbuvMbPDgaOBe4DRmQ1L\nRKRw7bFHWBNl3jxo2xZ69oQTToDS0vwc4JhOIinvHjoB+Ku7PwHUz1xIIiLFoUULuPrqcKvwySeH\n5q6uXcNaKfk0UWQ6ne2PA4uBY4ADCJ3u09x938yHVzPqIxGRfLZxIzz2GNxwAyxZEkbMZ7pjPlsr\nJDYCjgPecvd5ZrYLsI+7PxOn4ExQIhGRQvHKKyGhvPgiDB4cOuZ33LH2y8lKZ7u7fw3MB441swuA\nnXIxiYiIFJLDDgsd8y++CJ98AnvuCUOGwPz5SUf2fenctXUhMA7YKXr8w8yGZjowERGBH/wA7rwz\n3NnVrBkccgj07Quvv550ZJuk07Q1CzjM3b+KXjcGXnH3LlmIr0bUtCUihW7NGrjrrjDzcKdOcOml\ncNRRYdngrZGt9UiMTXduET2PVaiIiGydJk1CJ/z8+XDWWWGCyIMOCksCJzUFSzo1kkuAc4CHo02n\nAPe6+y0Zjq3GVCMRkWKzcSM8/niY32vZMhg2DM45J0wcmY5sTpFyAHB49HKqu78Rp9BMUSIRkWL2\n4otw3XVhaeCLLgp3ezVtWv1nMp5IzKwEmOPuneIUki1KJCIiYSr766+Hp58OgxwvvBB22qnyfTPe\nR+LuZcC7ZtYmTiEiIpI9XbrAuHEwbVqY06tTJxg6FBYtykx56XS2bw/MiVZHnFj+yEw4IiJSWzp0\ngNGjw63DjRrB/vuHkfLvvlu75aTT2d69su3u/p/aDSU+NW2JiFTt88/h9tvD4lsDB4ZtGe0jMbOO\nQEt3f6l4TDW9AAAJkklEQVTC9sOBpe6ec+MrlUhERGom030ktwCrK9n+RfSeiIhItYmkpbu/VXFj\ntK1dxiISEZG8Ul0iaVbNew1rOxAREclP1SWS18xsUMWNZnYeMCNzIYmISD6prrO9JWFalHVsShwH\nEVZHPNXdP8lKhDWgznYRkZrJ1sJWRwI/jF7OcffJcQrMJCUSEZGaydpcW/lCiUREpGayNY28iIhI\nlZRIREQkFiUSERGJpW5SBZvZH4HewFpgPjDQ3b83kt7MFhBG028E1rt712zGKSIi1UuyRvIM0Nnd\n9wPmAZdXsd9GoIe7768kIiKSexJLJO7+nLtvjF6+CrSuYldDTXAiIjkrV36gzwWerOI9B541s+mV\njbQXEZFkZbSPxMyeBVqmbiIkhivd/bFonysJfR/3V3GYbu6+1Mx2JCSUt939xarKHDFixHfPe/To\nQY8ePeKdhIhIASktLaW0tLRWj5nogEQzGwAMAnq6+9o09h8OrHH3m6p4XwMSRURqIK8HJJrZccAw\n4KSqkoiZNTKzbaPnjYFewOzsRSkiIluSWI3EzOYRJoBcGW161d3PN7NdgLvc/UQza0+YONIJzXDj\n3P26ao6pGomISA1orq0KlEhERGomr5u2RESkMCiRiIhILEokIiISixKJiIjEokQiIiKxKJGIiEgs\nSiQiIhKLEomIiMSiRCIiIrEokYiISCxKJCIiEosSiYiIxKJEIiIisSiRiIhILEokIiISixKJiIjE\nokQiIiKxKJGIiEgsSiQiIhKLEomIiMSiRCIiIrEokYiISCxKJCIiEosSiYiIxKJEIiIisSiRiIhI\nLEokIiISixKJiIjEokQiIiKxKJGIiEgsSiQiIhKLEomIiMSiRCIiIrEokYiISCxKJCIiEosSiYiI\nxKJEIiIisSiRiIhILEokIiISixKJiIjEokQiIiKxKJGIiEgsSiQiIhKLEomIiMSiRCIiIrEokYiI\nSCyJJRIz+6OZvW1mb5rZBDNrWsV+x5nZO2b2npldmu04RUSkeknWSJ4BOrv7fsA84PKKO5hZHeB2\n4FigM3CmmXXKapR5qrS0NOkQcoKuwya6FpvoWtSuxBKJuz/n7hujl68CrSvZrSswz90Xuvt6YDxw\ncrZizGf6HyXQddhE12ITXYvalSt9JOcCT1ayvRXwUcrrj6NtIiKSI+pm8uBm9izQMnUT4MCV7v5Y\ntM+VwHp3vz+TsYiISGaYuydXuNkAYBDQ093XVvL+ocAIdz8uen0Z4O5+fRXHS+5kRETylLtbnM9n\ntEZSHTM7DhgG/LiyJBKZDnQ0s7bAUqAvcGZVx4x7MUREpOaS7CO5DdgWeNbMXjezUQBmtouZPQ7g\n7mXABYQ7vOYA49397aQCFhGR70u0aUtERPJfrty1FUsxD1o0s9ZmNtnM5pjZW2b262j79mb2jJm9\na2ZPm9l2SceaLWZWJ6rlToxeF+W1MLPtzOzf0cDfOWZ2SBFfi8ujazDLzMaZWf1iuRZmdo+ZLTOz\nWSnbqjz36FrNi743vdIpI+8TiQYtsgG4xN07A4cBQ6Lzvwx4zt33BCZTyYDPAnYhMDfldbFei1uB\nSe6+F7Av8A5FeC2iPtZBwP7u3oXQN3wmxXMtxhB+H1NVeu5mtjfQB9gLOB4YZWZb7HvO+0RCkQ9a\ndPdP3P3N6PmXwNuEwZ0nA/dFu90HnJJMhNllZq2BnwB3p2wuumsRTTl0hLuPAXD3De7+BUV4LYDV\nwDqgsZnVBRoCiymSa+HuLwKfV9hc1bmfROiL3uDuCwizjnTdUhmFkEg0aDFiZu2A/QgzBbR092UQ\nkg2wU3KRZdXNhLsBUzv/ivFatAc+NbMxUTPfX82sEUV4Ldz9c+BGYBEhgXzh7s9RhNcixU5VnHvF\n39PFpPF7WgiJRAAz2xZ4ELgwqplUvIui4O+qMLMTgGVRDa266njBXwtC880BwB3ufgDwFaE5oxi/\nFx2Ai4G2wK6EmsnPKMJrUY1Y514IiWQx0CbldetoW9GIqusPAmPd/dFo8zIzaxm9vzOwPKn4sqgb\ncJKZfQD8E+hpZmOBT4rwWnwMfOTur0WvJxASSzF+Lw4CXnL3z6IhBQ8DP6I4r0W5qs59MbBbyn5p\n/Z4WQiL5btCimdUnDFqcmHBM2fY3YK6735qybSIwIHp+DvBoxQ8VGne/wt3buHsHwvdgsrv3Bx6j\n+K7FMuAjM/tBtOkowlisovteAO8Ch5rZNlHH8VGEmzGK6VoYm9fSqzr3iUDf6K629kBHYNoWD14I\n40iiUfK3EhLjPe5+XcIhZY2ZdQNeAN4iVE8duILwH/8Bwl8XC4E+7r4qqTizzcy6A//t7ieZ2Q4U\n4bUws30JNx3UAz4ABgIlFOe1GEb44SwD3gDOA5pQBNfCzO4HegDNgWXAcOAR4N9Ucu5mdjnwc2A9\noan8mS2WUQiJREREklMITVsiIpIgJRIREYlFiURERGJRIhERkViUSEREJBYlEhERiUWJRCRiZmui\nf9uaWZUrcW7lsS+v8PrF2jy+SJKUSEQ2KR9U1R7oV5MPmlnJFna5YrOC3A+vyfFFcpkSicj3XQsc\nHs2ae2G0UNYfzez/zOxNMxsEYfS8mb1gZo8Sph/BzB42s+nRImPnRduuBRpGxxsbbVtTXpiZ3RDt\nP9PM+qQce0rKwlRjU/a/zsxmR7H8MWtXRaQKdZMOQCQHXUY0vQpAlDhWufsh0XxuL5lZ+bQR+wOd\n3X1R9Hqgu68ys22A6WY2wd0vN7Mh0Sy85Tw69mlAF3ffx8x2ij7zn2if/YC9gU+iMn9EWJzqFHfv\nFH2+aaYugki6VCMR2bJewNlm9gbwf8AOwB7Re9NSkgjARWb2JmFNmNYp+1WlG2GmYtx9OVAKHJxy\n7KUe5jF6E2gHfAF8Y2Z3m9mpwDcxz00kNiUSkS0zYKi77x89do8WRoKwzkfYKUwU2RM4xN33I/z4\nb5NyjHTLKrc25XkZUDeaBr0rYdmAE4Gnanw2IrVMiURkk/If8TWEmWHLPQ2cH637gpntEa02WNF2\nwOfuvtbMOgGHpry3rvzzFcqaCvw06ofZETiCaqbtjspt5u5PAZcAXdI/PZHMUB+JyCbld23NAjZG\nTVn3uvut0TLGr0frWSyn8vW9nwJ+ZWZzCGtgvJLy3l+BWWY2I1ojxQHc/WEzOxSYCWwEhrn7cjPb\nq4rYmgKPRn0wEFb+E0mUppEXEZFY1LQlIiKxKJGIiEgsSiQiIhKLEomIiMSiRCIiIrEokYiISCxK\nJCIiEosSiYiIxPL/rTA3gwvtOREAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x118026fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_data(err_iteration, 'loss', [0, 100, -2, -4], 'Iterations', \n",
    "          'Cross entropy loss', 'Loss change over iterations')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Increasing Accuracy of Model over Training Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEZCAYAAAC5AHPcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu8VHW9//HXmzvKRQTEuIpgeUnFS0pRSekx6+Slu1qW\nZWaZHbsqWgaVRVR60nM8P7PM1J+XtCulmZpS6UmlBLwLIiIKgiAiKIJsPueP7xr3Ytyz9+zZe2b2\n5f18PNZjz6z1Xev7XWtmfz+zvt+1vksRgZmZWWv1qHcBzMysc3IAMTOzijiAmJlZRRxAzMysIg4g\nZmZWEQcQMzOriANIJyeph6T1kka3Z1prPUnLJL293uXoqCT9VNK0OpfhEUlT6lmGrqRXvQvQ3Uha\nDxRuvtke2AQ0ZPNOiYhrWrO9iNgKDGzvtGbtLSJOLryWdCjws4gYX638JF0JLIqIb+fKsHu18uuO\nHEBqLCJercAlPQ6cFBG3l0ovqWdENNSkcJ2Yj1PHUsbnIRp/SFVj+1YDbsKqL2VT4wzpO5KulXS1\npHXARyVNlvQPSWslPS3pAkk9s/Q9JW2VNDZ7f2W2/EZJL0i6U9K41qbNlr9b0qNZvhdKukPSx5vc\nkWbKmC3fW9ItktZIWi7pq7kynSPpMUnrJN0jaWdJEyRtLcrj74X8JZ0k6a9ZPmuAr0uaKOm2LI9V\nkq6QlA/YYyX9Nlu2StJ/SuqblfkNuXQ7S3pR0pAS+3qKpIezY3afpL1ziw/I5q2VdJWk3tk6O0q6\nIct3jaTZkkYW7duM7DN4IftMdsgt/6Skpdn6Z+Wby5ScnR3DVdl3Z3BTZc/Sf1bSIknPSvqNpBHZ\n/EskzSxK+0dJp2WvR2XpV0laLOnUXLrXfG+byPdKSd+UNAiYDYxValJ9QdKw5vaj8H2QdKKkpcCf\ns/nXS1oh6TlJtxc+R0mfAz4CnJ1t/9fZ/Pxx66v0vV6ezT9PUq9s2aGSlkj6WlaWpySdkNuX90p6\nKNv2k5JOL3W8u7SI8FSnCVgCvLNo3neAl4H3ZO/7AgcAbyIFm12AR4BTs+U9SU1gY7P3VwKrgP2y\nZdcCV1SQdifgBeC92bIvkZrbPl5iX5or4yDgGeA0oDcwADgwW3YWMA/YNXu/D7ADMAFoKMrj74X8\ngZOAV4DPZHn2BXYD3pGVd1iW/ge5fb8fmAX0z9K/OVt2MfCdXD5fBn5dYj+PA5YCk7L3E4FR2etl\nwP8Cw4EhwKPAp7Jlw4CjgT7Z/v8KuK5o3x4FdgX6AX8Dvp0t2zv7LA7Ojt/5wGbg7dnyr2Tr75xt\n/5LC59hE+Q/PPou9s7QXAX/Jlr0DeDyXdkfgxazsyj6nM7NjuSvwOPCOUt/bJvK+Evhm9vrQfF4t\n7Uf2fdgKXJodn75ZmT4ObJelvxCY21R+uXnLcsfte8Ad2X4OA+4CzsmV7xXgG9n+HglsAAZky1cB\nB2evdyh8H7rbVPcCdOeJ0gHk1hbW+wrwy+x1z+wfKx8U/ieX9kjgvgrSfhL4a1G+yykRQFoo48eA\nu0ukeww4oon55QSQx1oowwcK+QJvy8qvJtJNYduKcx5wTIlt3gp8rsSyZcCHcu/PAy4skfZAYGXR\nvp2Re/8FYHb2+lvA5bll22WVW6EiXAi8Lbd8DLCxRL6/AM7NvR8IbAFGkirkZcDkbNlngZtyx+ix\nom19A/hJK763LQWQkvtR+D6QBesS2x+Wfb/7F+dX9BkVjtsTwKG5Ze8BFubK90L++wKsAfbPXj8F\nfIosoHTXyX0gHdOy/JvstPw80q/87UiB4O5m1n8m9/ol0i/e1qYdWVwO0j9Nk1oo4xhgcYlVx5B+\nyVai+DiNIP0KnULaj56kX4oAo4EnIvvvz4uIOyVtUbo65/msTDc0U95S+wKwMvf6JWBUVrbtgQuA\nfwMGkyrr4s+lrM8iIl6StDaXdizwBzU2+QnYKmmniFjFtkYCd+a2tT7b1qiIWC7pOtJZ1l3A8cBP\nc3mMk/RcLo8eQL7/rvj70lol9yNX3qcLryX1AL5P+qEwlNSnEqRAUk5ZRgJP5t4vJfu8MquLvi/5\nz+R9pAD6I0nzgWkRcU8ZeXYp7gPpmIoruZ+Qml92jYjBwHSK+k6qYAWpsswb1VTCTHNlXEZq6mnK\nk6Rfl8VeBJDULzdv56I0xcdpFqkZZa+I2AE4sagM4ySVOm5XACdk03UR8UqJdMtKlLclZwDjSE13\nOwDvbMW6K0gBEHg1GOX7Z5YB/xYRO2bTkIjYvongAeksLN/PNTDbVqFivgb4kKRdSE2bv8nlsbAo\nj8ERcUxu263pFG8qbWv2A1Lz1RHA1OyYTmTbfsWWyrPNscheP10i7baFj5gbEUeTmitvIDX/djsO\nIJ3DQGBdRGyUtAdwSg3y/COwn6R/V+ro/iLpl10lZZwNjJF0qqQ+kgZKelO27FLgXEm7AkjaV9IO\nEfEM6Rf5x5TuX/kM2/6zlyrDi8B6SWOAr+aW/YPUBPE9Sf0l9ZP0ltzy/w98kPTr+4pm8vgZcIak\nSVl5J0pqLrAWDCD9gl0naSgpwJbreuAYSQcpdcp/m20rx58AM7N9RtJOko4ssa1rgJMkvVFSX2Am\n8LeIWA4QEf8E1pP6H26MiBez9f4BbJb05azzuWe2jf1bsR95K4FhkvJnYS3tR3HwH0jql1ubBdXv\nse1xWUnqqynlGuCbkoZKGk46o7iypYJn353jJA2MdCXYBlLzWrfjAFJf5f5i+wpwoqQXgP/Ha3/t\nRInXLeVZMm32q+8jwH8Cq4HxpL6BTa0tY0S8QGq6+SDpn/pRoHDD3Q+B3wF/ya7e+QmpkxTgZODr\nwLOkiuCuFvZtOqmj+flsm7/KlaGBdEHAnqRfuktJTR+F5U8ADwCbIqJkPhFxLelM55dZeX9N49lA\nc8f+fFJn6xpSx21xE1lzn8X9pIsYfkX6hfxstp3CZ3E+8Ccaj+EdpD6Wprb1Z1IA+l22rdG89oqp\na0h9AFfl1msg9REcROo7WEW6+KA19xW9uo8R8SDp2D2RXUE1rIz9KD5Gl5HOzpaTzn7vKFr+M2CS\n0lVv1zWxjW8BC0if+3xSkPx+OeUHPpGV/XlSf+FrrjrrDtREk3D7bVy6lPRPuzIi9imR5kLg3aRf\njidGxPxs/hHAj0lB7tKImFW1glqLsvbm5cAHIuLOltJ3RpJ+QerY/XZLaespa3Z6nnQxRFlNLmbV\nUO0zkMuAd5VaKOndwISI2I3U5HFxNr8H8N/ZunsBx0nyHaQ1JuldkgZnTR3fJF062iU7CrMmtKOB\nn9e7LE2RdGTW9DaA9Ev9Xw4eVm9VDSARcQewtpkkR5O1N0fE3cDg7Eqag0hDECzNOjOvzdJabb2V\ndIXUSlIT1DHNdC53WpK+R2qe+25ElLzSrM7eRzoDfJJ0tdJx9S2OWf2HMhnFtpfbPZXNa2r+QTUs\nlwERcQ5wTr3LUW0RcTZwdr3L0ZyI+BTpvgOzDqOjdaJX+9JUMzNrJ/U+A3mabe81GJ3N60M6TS+e\n3yRJ1bsSwMysi4qINv1or8UZyGsGDMyZTboZCEmTgecjYiUwF5goaZykPsCxWdqS6n1Lf0eZpk+f\nXvcydITJx8HHwsei+ak9VPUMRNLVwFRgqKQnSdfp9wEiIi6JiBslvUfSY6TLeD9JWtigNALozTRe\nxvtwNctqZmatU9UAEhHHl5HmtBLzbwLe0NQyMzOrv47WiW5tNHXq1HoXoUPwcWjkY9HIx6J9VfVO\n9FqRFF1hP8zMakUS0cZO9HpfhWVmVhW77LILS5curXcx6m7cuHE88cQTVdm2z0DMrEvKfmHXuxh1\nV+o4tMcZiPtAzMysIg4gZmZWEQcQMzOriAOImVmNjR8/nttuu63exWgzBxAzM6uIA4iZmVXEAcTM\nrE42b97MF7/4RUaNGsXo0aP50pe+xCuvpGe2rVmzhiOPPJIhQ4YwdOhQDjnkkFfXmzVrFqNHj2bQ\noEHsscce3H777XUpv28kNDOrk3PPPZd77rmH++67D4CjjjqKc889l29961ucd955jBkzhjVr1hAR\n3HXXXQAsXLiQiy66iH/961+MGDGCJ598koaGhrqU32cgZtYtSe0ztcXVV1/N9OnTGTp0KEOHDmX6\n9OlceeWVAPTu3ZsVK1awZMkSevbsyZQpUwDo2bMnmzdv5oEHHmDLli2MHTuW8ePHt/VwVMQBxMy6\npYj2mSpRuDt8+fLljB3b+Oy8cePGsXz5cgC+9rWvMWHCBA4//HAmTpzIrFmzAJgwYQI//vGPmTFj\nBiNGjOD4449nxYoVbT4elXAAMTOrA0mMGjVqm/G6li5dysiRIwEYMGAAP/rRj1i8eDGzZ8/m/PPP\nf7Wv49hjj+Xvf//7q+tOmzat9juAA4iZWc0VxqY69thjOffcc1m9ejWrV6/mO9/5DieccAIAN9xw\nA4sXLwZg4MCB9OrVix49erBw4UJuv/12Nm/eTJ8+fejfvz89etSnKncAMTOrMWWdJ+eccw4HHHAA\n++yzD/vuuy8HHnggX//61wFYtGgRhx12GAMHDmTKlCl8/vOf55BDDmHTpk1MmzaN4cOHM3LkSJ59\n9llmzpxZn/3oCqNVejReMyvm0XgTj8ZrZmYdjgOImZlVxAHEzMwq4gBiZmYVcQAxM7OKOICYmVlF\nPJiimXVJ48aNe/V+i+5s3LhxVdu27wMxM+uGfB+ImZnVjQOImZlVxAHEzMwq4gBiZmYVcQAxM7OK\nOICYmVlFHEDMzKwiDiBmZlYRBxAzM6uIA4iZmVXEAcTMzCriAGJmZhVxADEzs4o4gJiZWUUcQMzM\nrCIOIGZmVhEHEDMzq4gDiJmZVcQBxMzMKuIAYmZmFXEAMTOzilQ9gEg6QtIjkhZKOrOJ5TtI+o2k\nBZLukrRnbtkT2fx5ku6pdlnNzKx8iojqbVzqASwEDgWWA3OBYyPikVyaHwDrI+I7kt4AXBQRh2XL\nHgcOiIi1LeQT1dwPM7OuRhIRobZso9pnIAcBiyJiaUS8AlwLHF2UZk/gNoCIeBTYRdLwbJlqUEYz\nM6tAtSvnUcCy3Punsnl5C4D3A0g6CBgLjM6WBXCLpLmSTq5yWc3MrBV61bsAwPeBCyTdC9wPzAMa\nsmVTImJFdkZyi6SHI+KOpjYyY8aMV19PnTqVqVOnVrXQZmadyZw5c5gzZ067brPafSCTgRkRcUT2\nfhoQETGrmXWWAHtHxIai+dNJfSXnN7GO+0DMzFqhM/SBzAUmShonqQ9wLDA7n0DSYEm9s9cnA3+N\niA2StpM0IJu/PXA48ECVy2tmZmWqahNWRDRIOg24mRSsLo2IhyWdkhbHJcAewOWStgIPAidlq48A\nfispsnJeFRE3V7O8ZmZWvqo2YdWKm7DMzFqnMzRhmZlZF+UAYmZmFXEAMTOzijiAmJlZRRxAzMys\nIg4gZmZWEQcQMzOriAOImZlVpMUAIumXkt4lqU03nJiZWddSzhnIZcCngIWSzpU0scplMjOzTqDF\nABIRN0XER0gPh3oGuF3S3ySdIKkjDAdvZmZ1UFYfiKQhwPHACcB9wE+AtwA3Va9oZmbWkbU4mKKk\n64G9gauAyyLiqdyyeRGxX3WL2DIPpmhm1jrtMZhiOQHk34BbO3IN7QBiZtY6tRqNdwIwOJfpEEmf\naUumZmbW+ZVzBjI/IiYVzesQTVcFPgMxM2udWp2B9CzKtAfQuy2ZmplZ51fOZbi3SLoGuDh7/1ng\n1uoVyczMOoNymrB6AqcCh2azbgF+EhFbqly2srkJy8ysdWpyFVZn4ABiZtY67RFAWmzCkjQB+C6w\nJ9CvMD8iXt+WjM3MrHMrpxP9F6TxsAS8G7gO+GUVy2RmZp1AOX0g/4qIAyTdHxF7Z/P+GREH1qSE\nZXATlpl1FA0N8MorsGULbN2apgiQoEePNBXSFKaGhpS+oWHbbfXoAb17p6lnz7S8MBW2nc9j69a0\nntSYX2F+BAwbBq97XSFNDZqwgE3ZpbuLJX0WeBoY2JZMzazridi2Uty8GV5+GTZuTH83b26sKAsV\nXWG9fMVYqEgLrzdubNzGyy/Dpk1pW4VKOp+ukF9heSFNIV1xvoXKtTCVKl++Eo6AXr1ShS6lPF96\nKU2bNqV1evduTNOjR0pX2EZDQ5pfCAz5tIVtFuQDTWG9/FTYdiEwFQJHfr/yaT79aTj99Pb7zMs5\nAzkYeAgYQuoLGQT8ICLubL9itI3PQKyr27wZ1q+HF15I0/r1qcLq3Rv69IG+fVNlUahsXn4ZXnwR\nNmxIf/MVar5SL1TIhUo5XyHnNTRsW0Hn/27a1FjB9eqVylOoGPv3h3790tSnz2sr1YJ8pZivTHv1\nStvIb6dv38Y8CmmL0+XL0KtX6XyLK+By0hWORyHQ9O8P22+f/vbtm9bvDKp+FVZ2Ce93I2JaWzKp\nNgcQ6ygiUqVaqOhfeAGefx7WrUuVfuHX9EsvwerV8MwzaVq/fttfqIUKeuPGFAS2bIFBg2DgQBg8\nOL3u3z/NLwSAfHNH374p7YABsN1221ao/fq9tkIuVMr5yra4As1X0MXrFypqP3au86jVYIp3R8TB\nbcmk2hxArFIRqYJeuzZV9C+91PhrffPmNG3alCr0tWvhuedgzZoUEDZsSNP69WlZYRs9e6bKe+DA\nVNEPGbJtpV+ogIcPh513hhEjUtrCL99CZV2ooAcOTH9dOVt7qlUA+R9gZ+B64MXC/IiY3ZaM25MD\nSPe0ZUuqzAu/4p9/vrEJZvPmtGz16sZpzZoUANata/zVvmlT+gU9ZEia8r/WC01DffqkCnzIENhx\nxzTtsEP6dV+YdtyxcRt9+tT7yJi1rFYB5MomZkdEfLwtGbcnB5DOKSJV4IVmnhUr0rR8OTz1FCxb\nlqbVq7dtpy/8lWDo0PQLfuedU6VeuFqld+9UqQ8fnq48GTYspR06NJ0JFNrJC80vZt2N70TPOIDU\nV0MDLF0KDz+cppUrGyv5Qlt/Ydq4sbEzd+PGtP6gQamJ53Wvg5Ej09/Ro2HMGBg7NlX+hXb6/NRZ\nOivNOqJa3Yl+SVPzI8LPBOkm1q+HRYtg8WJ4/PE0PfEErFqVpmefTWcAe+yRppEjGyv5/v0bzwCG\nDUvv8525/fq1mL2ZdVDl3Afyl9zrfsD7gGXVKY7V07p1KUgsWADz56e/jz6a5k+cCBMmpGnSJDj6\n6NR0tNNOqZnIgcCs+2l1E1Z2U+EdEfGW6hSp9dyEVZ4NG1JT09KlKVA89liali6FJ59Ml5COHw/7\n7puCxL77wu67w6hRjde/m1nXUKs70YuNB0a0JVOrroh0VdI//wl33ZWm+fNTn8PYsTBuXDqTmDgR\n3vnOFDTGjEmd0L5U1MzKVc5VWGuBQqIewHPAtIi4rsplK1t3PgNZswYefBAeeij9feABuP/+FEQO\nPBAmT07T/vun5iYHCDOD2l3Gm7/WZWtHrKm7SwB5+eV0NvGPf8DcuWlavx723BP22iv9feMbYe+9\nU/+Eg4WZlVKrAHIU8NeIWJe93wF4a0T8sS0Zt6euHEAWL4brr4dbb4W7706BYsoUeNOb0rTrrg4U\nZtZ6tQog8yNiUtG8eRGxX1sybk9dLYA88wxcdx1cfTUsWQIf/CAccQS8/e3pfgkzs7aqVSd6UxlU\n0vluzVixAn7zm3S2sWABvPe9MGMGHHZYGqTOzKyjKecM5BfAKuCibNZpwAgPZdI2Eakv48Yb07Rk\nSQoaH/oQHH6476sws+qqVRPWAGAGcBjpaqxbgG9HxIa2ZNyeOlMAWbUKLr8cfvrTdGZxzDHw7nfD\nm9/sMw0zqx2PhZXp6AEkInWAX3hhOts45hg45ZR0ea07wM2sHtojgLR4f7Gkm7Irrwrvh0i6oS2Z\ndhcvvwxXXAEHHQQf/Wi6L2PJEvjFL9IZh4OHmXVm5TSajIiI5wtvImKtpJFVLFOn9/jjcPHFcNll\ncMABMH16aqby6LFm1pWUM8LRVkmjC28kja1ieTq1uXPhAx9IZxxbt6ZO8ptuSp3jDh5m1tWUcwby\nTeBOSbeRLumdCpxazUJ1Nn/7G3zrW2nI869+NTVbbb99vUtlZlZdZXWiSxoBvDl7+78RsaqqpWql\nenWiL1oEZ5wB8+alAHL88X66nZl1DjXpRAeIiJUR8TtgPnCSpAXlZiDpCEmPSFoo6cwmlu8g6TeS\nFki6S9Ke5a5bLxs3wle+kjrCJ0+GRx6BT3zCwcPMupdyrsIaIekLkv4BPAJsB5xYzsazZ4f8N/Au\nYC/gOEm7FyU7G5gXEfsCnwAubMW6NXfvvaljfPnyNALumWf6pj8z655KBhBJn5J0CzAH2JnU77Ei\nIs6JiHllbv8gYFFELI2IV4BrgaOL0uwJ3AYQEY8Cu0gaXua6NbN1K8ycmcak+sY34Jpr0vDoZmbd\nVXOd6D8B/gR8ICIegtTX0Mrtj2Lbx98+RQoMeQuA95M66g8CxgKjy1y3Jtavh499DJ57Lj2kaayv\nQzMzazaAjAI+DPxU0iDgeqAarfzfBy6QdC9wPzAPaGjtRmbMmPHq66lTpzJ16tR2KdzSpXDkkXDw\nwWmgwz592mWzZmY1NWfOHObMmdOu2yz3KqxxwLHAcUBP4LcR8c0y1psMzIiII7L304CIiFnNrLME\n2Bt4Y7nrVusqrHvuScOOnHEGnH667xw3s66jlldhLY2IWdlzQT7Siu3PBSZKGiepDykIzc4nkDRY\nUu/s9cmkh1dtKGfdanrooXTmcfHF8MUvOniYmRVr9fivWX9Ii2cfWdoGSacBN5OC1aUR8bCkU9Li\nuATYA7hc0lbgQeCk5tZtbXkrsWxZGnrkRz+Co46qRY5mZp2PR+Mt8txz8La3wSc/me4qNzPrijyc\ne6a9AsiLL6aHOU2eDOed1w4FMzProGr1QKl9mpi9DlgWEVvbknl7aY8AsmlT6vMYORJ+/nPoUVbv\nkJlZ51SrADIXmETqnxCpz+IhYCDwmYj4S1sK0B7aGkC2bIEPfzgFjWuv9ZMBzazrq9VVWE8AB0TE\npGy4kQOAhaQhRjp9Q8/WrXDSSfDSS3DVVQ4eZmblKqe63CMi7iu8iYj7Je0ZEY+pC1zbOnNmGlX3\n1luhb996l8bMrPMoJ4A8Ium/SGNRQboP5BFJfYEtVStZDdx+O1x0UXoQ1Hbb1bs0ZmadSzl9INsB\nXwDems26E/gv4GVgQESsq2oJy1BJH8iKFWlU3SuugMMOq1LBzMw6KF/Gm2ltANmyBQ49NE3fLOuW\nSDOzrqU9AkiLTVjZeFbTgXH59BHx+rZkXE+zZqX+jq9/vd4lMTPrvMppwnoYOAP4F7lRciNiZXWL\nVr7WnIGsWQNveEPq9xg/vsoFMzProGpyBgK8EBF/aEsmHckPfwgf/KCDh5lZW5VzBjIze/kbYFNh\nfv7S3nor9wxk5UrYc0+YPx/GjKlBwczMOqha3Yn+9yZmR0S8vS0Zt6dyA8iXvpRuHLzgghoUysys\nA/NVWJlyAshTT8E++6TnfOy8c40KZmbWQVU1gEg6LiKukfQfTS2PiAvbknF7KieAnHoqDBgAP/hB\njQplZtaBVbsTfUj2d3hbMugINmxI41w99li9S2Jm1nV0iyasa6+Fyy+HP/2phoUyM+vAanUj4TDg\nU8AubHsj4WfaknEtXX99Gq7dzMzaTzlXYd0J3MVrbyT8ZXWLVr7mzkA2bIBRo2DJEthxxxoXzMys\ng6rVjYTbR8RX2pJJPf3xj/CWtzh4mJm1t3IeKPUnSYdXvSRVcv318KEP1bsUZmZdTzlNWGuBwcBL\nwGbSY20jIjrMb/pSTVhuvjIza1qtmrCGtSWDerrhBjdfmZlVS8kAImm3iFgE7FUiSYcZC6sUN1+Z\nmVVPc3eiXxoRJ3XWsbDcfGVmVlpVm7Ai4qTs79vakkG93HEH7L+/g4eZWbWU0weCpN2BPYF+hXkR\ncXW1CtUe7r0X3vSmepfCzKzrKudO9G8AhwO7A38G3gXcAXToADJvHrz//fUuhZlZ11XOfSAfAd4B\nrIiIE4B9ge2rWqp2MG8e7LdfvUthZtZ1lRNANkZEA7BF0kDgGWBcdYvVNuvWpacP7rZbvUtiZtZ1\nldMHMk/SDsDPgX8CLwD3VLVUbTR/Puy9N/TsWe+SmJl1Xc0GEEkCZkTE88BFkv4MDIqIe2tSugq5\n+crMrPqabcLKbq64Jff+sY4ePMABxMysFsrpA5kvqVNVxw4gZmbV19yd6L0iYoukB4E3AIuBF2kc\nTHH/2hWzefk70V9+Od08uHYt9O1b54KZmXVQ1R5M8R5gf+CotmRQaw88kK6+cvAwM6uu5gKIACJi\ncY3K0i7cfGVmVhvNBZDhkr5camFEnF+F8rTZvHlpDCwzM6uu5jrRewIDgIElpg7JZyBmZrXRXCf6\nvR2po7w5hU70hgYYPBiWL4dBg+pdKjOzjqs9OtGbOwNp04br4dFH4XWvc/AwM6uF5gLIoTUrRTtx\n85WZWe2UDCAR8VwtC9IeFiyASZPqXQozs+6hnDvRO43ly2Hs2HqXwsyse+hSAWT1ahg2rN6lMDPr\nHhxAzMysIl0qgDz7rAOImVmtVD2ASDpC0iOSFko6s4nlQyX9SdJ8SfdLOjG37AlJCyTNk9TiQ6x8\nBmJmVjslbyRsl41LPYCFpEuClwNzgWMj4pFcmulAv4g4S9Iw4FFgRDYS8OPAARGxtoV84sUXg6FD\n4aWXQJ3uDhYzs9qq9o2E7eEgYFFELI2IV4BrgaOL0jxD49AoA4E1EbEle69yy7hmTTr7cPAwM6uN\nageQUcCy3Punsnl5PwX2krQcWACcnlsWwC2S5ko6ubmM3HxlZlZbzT4TvUbOAhZExDskTSAFjH0i\nYgMwJSJWSBqezX84Iu5oaiPnnz+DdetgxgyYOnUqU6dOrd0emJl1cHPmzGHOnDntus1q94FMBmZE\nxBHZ+2mkpxnOyqW5EfhuRNyZvf8LcGZE/LNoW9OB9U0NIy8prroq+MMf4JprqrY7ZmZdRmfoA5kL\nTJQ0TlIf4FhgdlGah4HDACSNAF4PPC5pO0kDsvnbA4cDD5TKyE1YZma1VdUmrIhokHQacDMpWF0a\nEQ9LOiUT34tFAAAIP0lEQVQtjkuAmcBlkhaQOs3PiIjnJI0HfispsnJeFRE3l8pr9WoYPryae2Nm\nZnlVbcKqFUnxuc8Fb3wjnHpqvUtjZtbxdYYmrJpxE5aZWW11mQDiYUzMzGqrywQQn4GYmdWWA4iZ\nmVWky3Si9+4drF8PffvWuzRmZh2fO9Fz+vd38DAzq6UuE0DcfGVmVlsOIGZmVhEHEDMzq4gDiJmZ\nVcQBxMzMKtJlAogHUjQzq60uE0B8BmJmVlsOIGZmVhEHEDMzq4gDiJmZVcQBxMzMKtJlBlPcsiXo\n2bPeJTEz6xw8mGKOg4eZWW11mQBiZma15QBiZmYVcQAxM7OKOICYmVlFHEDMzKwiDiBmZlYRBxAz\nM6uIA4iZmVXEAcTMzCriAGJmZhVxADEzs4o4gJiZWUUcQMzMrCIOIGZmVhEHEDMzq4gDiJmZVcQB\nxMzMKuIAYmZmFXEAMTOzijiAmJlZRRxAzMysIg4gZmZWEQcQMzOriAOImZlVxAHEzMwq4gBiZmYV\ncQAxM7OKVD2ASDpC0iOSFko6s4nlQyX9SdJ8SfdLOrHcdc3MrH6qGkAk9QD+G3gXsBdwnKTdi5Kd\nBsyPiEnAO4DzJPUqc10rMmfOnHoXoUPwcWjkY9HIx6J9VfsM5CBgUUQsjYhXgGuBo4vSPAMMzF4P\nBNZExJYy17Ui/gdJfBwa+Vg08rFoX9UOIKOAZbn3T2Xz8n4K7CVpObAAOL0V65qZWZ10hE70s4AF\nETES2A+4SNKAOpfJzMxaoIio3salycCMiDgiez8NiIiYlUtzI/DdiLgze/8X4EygV0vr5rZRvZ0w\nM+uiIkJtWb9XexWkhLnAREnjgBXAscBxRWkeBg4D7pQ0Ang98Diwrox1gbYfBDMza72qBpCIaJB0\nGnAzqbns0oh4WNIpaXFcAswELpO0ABBwRkQ8B9DUutUsr5mZla+qTVhmZtZ1dYRO9Ip15xsNJY2W\ndJukB7MbMP8jmz9E0s2SHpX0Z0mD613WWpHUQ9K9kmZn77vlsZA0WNL1kh7Ovh8Hd+NjcVZ2DO6T\ndJWkPt3lWEi6VNJKSffl5pXc9+xYLcq+N4eXk0enDSC+0ZAtwJcjYi/gzcDns/2fBtwaEW8AbiNd\n5dZdnA48lHvfXY/FBcCNEbEHsC/wCN3wWGT9pycD+0XEPqQm++PoPsfiMlL9mNfkvkvaE/gwsAfw\nbuB/JLXYt9xpAwjd/EbDiHgmIuZnrzeQLkYYTToGl2fJLgeOqU8Ja0vSaOA9wM9ys7vdsZA0CHhb\nRFwGEBFbImId3fBYAC8Am4HtJfUC+gNP002ORUTcAawtml1q348Crs2+L08Ai0h1bLM6cwDxjYYZ\nSbsAk4C7gBERsRJSkAF2ql/Jauo/ga8B+U697ngsxgOrJV2WNeddImk7uuGxiIi1wHnAk6TAsS4i\nbqUbHoucnUrse3F9+jRl1KedOYAYkN10+Svg9OxMpPiqiC5/lYSkfwdWZmdkzZ12d/ljQWqm2R+4\nKCL2B14kNVt0x+/FrsCXgHHASNKZyEfphseiGW3a984cQJ4Gxubej87mdRvZafmvgCsj4vfZ7JXZ\n/TRI2hlYVa/y1dAU4ChJjwPXAO+UdCXwTDc8Fk8ByyLin9n7X5MCSnf8XhwI3BkRz0VEA/Bb4C10\nz2NRUGrfnwbG5NKVVZ925gDy6k2KkvqQbjScXecy1drPgYci4oLcvNnAidnrTwC/L16pq4mIsyNi\nbETsSvoe3BYRJwB/oPsdi5XAMkmvz2YdCjxIN/xeAI8CkyX1yzqEDyVdZNGdjoXY9qy81L7PBo7N\nrlIbD0wE7mlx4535PhBJR5CuOCncaPj9OhepZiRNAf4G3E86DQ3gbNKHfh3p18RS4MMR8Xy9yllr\nkg4BvhIRR0nakW54LCTtS7qYoDdpVIdPAj3pnsfia6QKswGYB3yaNOp3lz8Wkq4GpgJDgZXAdOB3\nwPU0se+SzgJOAl4hNYnf3GIenTmAmJlZ/XTmJiwzM6sjBxAzM6uIA4iZmVXEAcTMzCriAGJmZhVx\nADEzs4o4gFi3J2l99necpCafetmGbZ9V9P6O9ty+WT05gJg1jgc0Hji+NStK6tlCkrO3ySjira3Z\nvllH5gBi1mgm8NZsFNvTswdU/UDS3ZLmSzoZ0t3ukv4m6fekYUKQ9FtJc7OHe306mzcT6J9t78ps\n3vpCZpJ+mKVfIOnDuW3fnnsg1JW59N+X9EBWlh/U7KiYlVDVZ6KbdTLTyIZBAcgCxvMRcXA23tqd\nkgrDO+wH7BURT2bvPxkRz0vqB8yV9OuIOEvS57NRcQsi2/YHgH0iYm9JO2Xr/DVLMwnYE3gmy/Mt\npIdCHRMRu2frD6rWQTArl89AzEo7HPi4pHnA3cCOwG7ZsntywQPgi5Lmk57JMjqXrpQppJGDiYhV\nwBzgTbltr4g0ztB8YBdgHbBR0s8kvQ/Y2MZ9M2szBxCz0gR8ISL2y6YJ2QOJID1nIyVKAzi+Ezg4\nIiaRKv1+uW2Um1fBptzrBqBXNhz5QaTh+98L3NTqvTFrZw4gZo2V93rSSK0FfwZOzZ67gqTdsqf7\nFRsMrI2ITdlz6Sfnlm0urF+U19+Bj2T9LMOBt9HM8NlZvjtExE3Al4F9yt89s+pwH4hZ41VY9wFb\nsyarX0TEBdnjgu/Nniexiqafn30T8FlJD5KeQfGP3LJLgPsk/St7RkkARMRvJU0GFgBbga9FxCpJ\ne5Qo2yDg91kfC6Qn7ZnVlYdzNzOzirgJy8zMKuIAYmZmFXEAMTOzijiAmJlZRRxAzMysIg4gZmZW\nEQcQMzOriAOImZlV5P8A+78HhVE1hHIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1181b4c90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_data(train_accuracy, 'loss', [0, 100, 0.8, 1], 'Iterations', \n",
    "          'Training Accuracy', \n",
    "          'Training accuracy change over iterations')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Increasing Accuracy of Model over Validation Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEZCAYAAAC5AHPcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYXFW57/HvrzMRyBwhkoSEJAgCyigBBSQIIg6ARz0a\nQGVQjkdB0asoqNwEQREVhXPE60EBEQgoIEdUUKYEBAXCFGYSIQlDQoCQQICETO/9Y62id5oeqqu7\nqnr4fZ6nnq49v3vX7vXuvdYeFBGYmZm1V0O9AzAzs+7JCcTMzCriBGJmZhVxAjEzs4o4gZiZWUWc\nQMzMrCJOIF2YpPGS1ktqyN3XSPpMOeNWsKyTJJ3bkXiteZJmSjq63nF0VV1h32vtf8ta1rfeAfRk\nkq4F7oiI6U36HwL8EhgTEevbmM0bN+pExIfKHbeNuPYBLo6ILQrzPr2cac06W3HfkzQemA/0LeN/\noyKSpgGTIuKzhRja+t+yZvgMpLouBD7dTP9PAxdV6x+kDKLMZNPdSepT7xisURm/R2nfVJXmb50p\nIvyp0gfYCFgG7FXoNwxYCbwjd38IuAd4CVgITCuMOx5YBzTk7pnA0fl7A/AT4HngX8CXmox7JPAw\n8HIe/h+5/8bAa8BaYEUe/lZgGimplZZ9MPAg8CJwE/D2wrD5wNeBOXn9LgX6t7ANJgI3Ai8AzwEX\nA0MKw8cCV+ZhzwP/VRh2TGEdHgR2yv3XAxML410AfC9/3wd4CvgmsJiUxIcBf8rLWJq/jy5MPxw4\nH3gmD/9D7v8A8OHCeH1zjDu2sK6HAPfm33IecEDhd/secGtel78CIwrT/T7HugyYBWzXZN1+Dvw5\nT/tPYEJh+AHAo3nac/L0RxeGH5234VLgWmBcK/trs7953paXNxn3bOCs/H0I8GtgUd72pwLKw47I\n6/3TvA98r5nlTgN+m78vJO3HpX1z97bWI+8PXwLmAo/nfmcBT+bfYjb5fxD4APB6/qwA7m3mf0vA\nd4EFwLPAb8j7LOl/cj3w2Rzrc8C3C7Hslpf3Uv5Nf1LvcqiqZVy9A+jpH+Bc4NxC9xeAewrd7wW2\nz9/fkXe6g3N3awnkP/M/1GhSAXlTk3E/CGyZv+8NvEpjAbwP8GSTOIv/xFsDrwDvA/oAJ5AKxL55\n+HzgdmBUXvbD5ATVzPpPAvYjFb4jSQXcT/OwBuA+UiLcCOgPvCcP+3dSYbRL7p4IbJG/r6P1BLIG\n+AHQDxgAjAD+LX/fBPgdcFVh+r+QkuCQvL575/4nAJcVxjsEmNPCek4GlgPvy92bA1sXfrd5eVsM\nyN0/KEx7JCmx9yMVtPc2WbfngV3z9roYmJGHjSQVVIfkYV8hFYxHF+Kdm3/PBuDbwG0txN/ibw6M\ny8M2Kfxui4DdcvdVwC/yb/iWvG8ck4cdkX+PL+XpBjSz7OK+V9rn1WS7t7gepAL9b8DQ0vyBw0j7\nZgPwNdL/Vf+myyvMo/i/dXRe3vj8u1zZJL71wP+Q9tcdgFXANnn4P4DD8/eNgcn1LoOqWr7VO4Ce\n/gH2JB0dlnbeW4HjWxn/Z8CZ+XtrCeRGCoU28P7iuM3M9yrgy/l7Wwnku2xYcAp4Gnhv7p4PHFoY\nfgbwizK3xyHA3fn7u4ElzcVMOkr/cgvzaOsMZBXQr5UYdgKW5u+bk87GhjQz3uako9RBufty4Bst\nzPOXpd+tmWEz2fAo9YvANS2MOyyv3+DCuhUPQD4IPJy/f4YmCYF01F3aR64BjioMayAdSGzRzHLb\n+s1vAT5d2Nfm5e+j8vYeUJh2KnBT/n4EsKCNfaK5BNJQGN7qeuTttU8by3gReGfT5TX5jUrb7Qbg\nPwvDtgZW5+WW4tu8MPwO4JP5+6w8/5Hl/D9094/bQKosIm4jHUF+VNJE0inujNJwSZMl3STpOUnL\nSWcobylj1qNJR+glC4sDJX1Q0j8lLZW0jFTwlDPf0rzfmF+k/4yngDGFcZYUvr8GDGpuRpI2k3Sp\npKfz+l1ciGMssDCabwvaAni8zHibej4i1hRiGCjpfyQtyDHcDAyTpBzDixHxctOZRMRiUsL/uKSh\npG14SQvLbCveZwvf39hekhok/VDSv3Js80ltAG9pa1revA9AKvRLxgNnS3pR0ouk6p9gw9+xpK3f\n/FLg0Pz9UBr34XGkM6fFeTnLSMm0GH/TGNurnPUorjeSviHpYUnLckxDqHD/z9/7kpJlSUv7/+eA\nbYBHJd0h6cNlLrNbcgKpjYtIR2KfBv4WEc8Xhs0A/pd0RdYw0qlxOQ2Ii0mFVsn40hdJ/YErgB8B\nm0bEcFK9cWm+0ca8FxXnl21Bk3/SMv2AdIS4fV6/TxfieAoY18Klx0+Rqnya8xqpeqDkrU2GN12/\nrwNvI1W5DCNVG5LjeAoYIWlIC8v6LelI/9+Bf+Sk0pzW4m3N4cBBpKqvYcCWOa5K9gFICbEY0xci\nYkT+DI+IQRFxezPzauk3fyZ/vxyYImkMqTqwlECeIp2BjCwsY1hE7FCYT1v7W1Fz4z5Zxnq8MZ2k\nvUhVcJ/I4w4ntadUuv+PJ1XDLWl+9EIQEY9HxGERsSnp/+8KSQPbmq67cgKpjd8C+wOfJzXqFg0C\nlkXEGkmTSXW3RS0VJL8HviJpjKThwLcKw/rnzwsRsV7SB0mNrSVLgJGtFJq/Bz4saV9JfSV9g1RI\n/LP11WzWYFL9+Ypc+JxQGHYnqRD8oaSNJQ2Q9J487NfANyTtAiBpkqRSYXkvcFg+ej+QVG3VVgwr\ngZcljQCmlwZExLOk5PoLScPy+u5dmPYqYBdS+8JvW1nGecBReZtJ0mhJW7cRF6Tf/3VgmaRNgNMp\nv8D9C/AOSQdL6iPpODY8Sv4l8G1J2wFIGirpEy3Mq6Xf/B8AEfEC6cztAuCJiHgs938WuA74maTB\ned0nSnpv84tp0/OkA45iMv6fdqwHpN97DbBUUn9J/zf3K1kCbJnPQJtzKfA1SVtKGgR8n1S9VzpT\nbjG5SzpcUulM5yXSb1mvqy2rzgmkBiJiIekfcWPg6iaDvwScKuklUj3075pO3sL3X5EaDucAd5Ea\n+krLe4VU4F2eT/mnAn8sDH+M9E/yRK4W2OAIPiLmks4Ufk76h/4wcFBErG0mjracQmoAXk66+qkY\n53rS0ffbSEeZTwGfzMOuIP3jzpD0MqkgH5En/SrpiqFlpOqUq9qI4SzStn+B9Dtc02T4Z0jtII+S\nCpfjCzGuAv4ATMh/mxURs4Gj8rJeItWFl45iW9tevyWt+zOkK6D+0ca6FJe5lHRm9GPSur2dtC+8\nnof/L/BD4LJcPXY/cGAL82rrN4d01rEfb67G+yzpgOVhUlvD5bz5rLDcdVpJ+t1vy/vm5DLWo+n2\n/Vv+zCVVCb7GhtVol5OSwFJJdzUzj/NJtQa3kKolXyP9P7W0vGL3gcBDeZ/9GfCpiHi9zRXvpkqX\n2lVn5tJ5wEeAJU1OaYvj/BepbvlV4MiIuC/3P5D0z9gAnBcRZ1QtULNWSPou6Yqqz7Y5ch3lI+qn\ngcMi4uZ6x2M9X7XPQC4gXXfdrFy1Miki3kZqPP5l7t9AOhL6ALA9cKikt1c5VrM3yVVenyNdjt3l\nSDogV+kMAL6TezfXxmHW6aqaQCLiVlI1Q0sOIdcrR8QdwFBJo0jX1M+LiIX5aprL8rhmNSPp86Tq\npWvyvtwVvZtUzfIcqdrpkJ5cZWJdS72fhTWGDesmn879mus/uYZxmRERvyY15ndZEXEKqZ3JrOa6\nWiN6Rc+/MTOz2qv3GcgzbHgd+9jcrz/pBqWm/ZslqXpXApiZ9VAR0aGD9lqcgbR2U9TVpEsAkbQH\nsDwilpAeRraV0jsu+pMuQ216+esG6n1Lf1f5TJs2re4xdIWPt4O3hbdF65/OUNUzEEkzgCmkm9ae\nJD0jpj/pSQnnRsQ1kj4k6V+ky3iPIg1cl2+Kuo7Gy3gfqWasZmbWPlVNIBHR9K7q5sY5roX+fyU9\nU8bMzLqgrtaIbh00ZcqUeofQJXg7NPK2aORt0bmqeid6rUiKnrAeZma1IonoYCN6va/CMjOrii23\n3JKFCxe2PWIPN378eBYsWFCVefsMxMx6pHyEXe8w6q6l7dAZZyBuAzEzs4o4gZiZWUWcQMzMrCJO\nIGZmNTZhwgRuuummeofRYU4gZmZWEScQMzOriBOImVmdrF69mq9+9auMGTOGsWPH8rWvfY01a9YA\nsHTpUg466CCGDx/OyJEj2Weffd6Y7owzzmDs2LEMGTKEbbfdlpkzZ9Ylft9IaGZWJ6eddhp33nkn\n999/PwAHH3wwp512GqeccgpnnnkmW2yxBUuXLiUiuP329KbiuXPncs4553D33XczatQonnzySdat\nW1eX+H0GYma9ktQ5n46YMWMG06ZNY+TIkYwcOZJp06Zx0UUXAdCvXz8WL17M/Pnz6dOnD3vuuScA\nffr0YfXq1Tz44IOsXbuWcePGMWHChI5ujoo4gZhZrxTROZ9KlO4OX7RoEePGNb47b/z48SxatAiA\nE044gUmTJnHAAQew1VZbccYZZwAwadIkzjrrLKZPn86oUaM47LDDWLx4cYe3RyWcQMzM6kASY8aM\n2eB5XQsXLmT06NEADBo0iJ/85Cc8/vjjXH311fz0pz99o61j6tSp/P3vf39j2hNPPLH2K4ATiJlZ\nzZWeTTV16lROO+00XnjhBV544QVOPfVUPvOZzwDwl7/8hccffxyAwYMH07dvXxoaGpg7dy4zZ85k\n9erV9O/fn4EDB9LQUJ+i3AnEzKzGlBtPTj75ZHbddVd22GEHdtxxR971rnfxne98B4B58+ax//77\nM3jwYPbcc0+OPfZY9tlnH15//XVOPPFENt10U0aPHs3zzz/P6aefXp/16AlPq/TTeM2sKT+NN/HT\neM3MrMtxAjEzs4o4gZiZWUWcQMzMrCJOIGZmVhEnEDMzq4gfpmhmPdL48ePfuN+iNxs/fnzV5u37\nQMzMeiHfB2JmZnXjBGJmZhVxAjEzs4o4gZiZWUWcQMzMrCJOIGZmVhEnEDMzq4gTiJmZVcQJxMzM\nKuIEYmZmFXECMTOzijiBmJlZRZxAzMysIk4gZmZWEScQMzOriBOImZlVxAnEzMwq4gRiZmYVcQIx\nM7OKOIGYmVlFnEDMzKwiVU8gkg6U9KikuZK+1czwYZL+IGmOpNslbVcYtiD3v1fSndWO1czMyqeI\nqN7MpQZgLrAfsAiYDUyNiEcL4/wIWBERp0raBjgnIvbPw54Ado2IZW0sJ6q5HmZmPY0kIkIdmUe1\nz0AmA/MiYmFErAEuAw5pMs52wE0AEfEYsKWkTfMw1SBGMzOrQLUL5zHAU4Xup3O/ojnAxwAkTQbG\nAWPzsACulzRb0jFVjtXMzNqhb70DAH4InC3pHuAB4F5gXR62Z0Qszmck10t6JCJubW4m06dPf+P7\nlClTmDJlSlWDNjPrTmbNmsWsWbM6dZ7VbgPZA5geEQfm7hOBiIgzWplmPvDOiHilSf9ppLaSnzYz\njdtAzMzaoTu0gcwGtpI0XlJ/YCpwdXEESUMl9cvfjwFujohXJG0saVDuvwlwAPBgleM1M7MyVbUK\nKyLWSToOuI6UrM6LiEckfSENjnOBbYELJa0HHgI+lycfBVwlKXKcl0TEddWM18zMylfVKqxacRWW\nmVn7dIcqLDMz66GcQMzMrCJtJhBJfWoRiJmZdS/lnIHMk/Tj4jOqzMzMykkgO5KeZ/Xr/LDD/5A0\npMpxmZlZF9euq7Ak7QPMAIYBVwCnRsS/qhRb2XwVlplZ+9TkKixJfSQdLOkq4CzgTGAi8Cfgmo4s\n3MzMuq9ybiScB8wEfhwR/yj0v0LSe6sTlpmZdXVtVmFJGtT0uVRdjauwzMzap1Y3Ep4jaVhhocMl\nnd+RhZqZWfdXTgLZISKWlzry2wF3rl5IZmbWHZSTQBokDS91SBpB13iPiJmZ1VE5ieBM4J+SLie9\nYvYTwPerGpWZmXV5Zd0HIml7YN/ceVNEPFzVqNrJjehmZu3TGY3oZd9IKGkzYKNSd0Q82ZEFdyYn\nEDOz9qnVjYQHS5oHzAduBhYA13ZkoWZm1v2V04h+KrAHMDciJgD7AbdXNSozM+vyykkgayJiKelq\nrIaImAm8q8pxmZlZF1fOVVjLJQ0CbgEukfQc8Gp1wzIzs66unEeZbAKsJJ2tHA4MBS7JZyVdghvR\nzczap+pXYeW3Ed4QEfu2OFIX4ARiZtY+nZFAWq3Cioh1ktZLGhoRL3VkQWZm3U0ELF8OL7wAy5al\nz+uvw0YbpU///vDyy43D1qxpnHbtWli1ClauTNMUj3H79oWBA9Onf3+QGpe3alXjdP36wfDh6TNo\nUOOwVas2nF+R1BjfRhtBn8JLybfeGrbfvvO2TzltIK8AD0i6nkLbR0R8pfPCMLOubM0aePZZWLwY\nliyBdetS/4g0rFTgFQtQgAEDUiE2cGAqREsF7UsvNV8YRqSCd+XK9Fm7Ns1j4MA3F4ZFa9c2znv5\ncmhoaJxG2rAgb0mfPo2x9ukDzz2X1rd/f3jLW1IhPmJEiqcU9+rVMGRIGjZsWBpWUoph4MA0vKFw\nydKaNSmel156c0ylGAYPTvNfsADuvRdefbVx2IABG86vaP36Dbft+vWNwz72sdonkD/kj5l1Y+vX\np6NlSAVRqbArFdivvAILF8Ljj8MTT2z4ee452HRTGD0aRo1KR9Al/fo1Ftb9+m14NL16dWMy6N+/\n8Wh6s80aC9emhWG/fo0FZd++qYBdufLNhWFRnz6pAC8V5BGN00Q0xjdgQGN8TRXPGNauTeu5+eaw\n8cYd2+49WbteadtVuQ3E2rJmTToyLR2lFo+ES4VS6Wi6VPCsXbvhEV+pMCwVMqUjvNJ4pU9LR4ZF\nEalgLM1rxYrGmF57DYYObSwMS0eUK1emeZcK4dIRajGeUmFd7Pfaa2ndV6yATTZJBWjpbEFK89xo\nozRs/HiYOBEmTYIJE9LfiRNhzJiWj/6te6rJo0wkzQfeNFJETOzIgjuTE0j3FpEK8tKRZ58+6XR9\n0aJUhVA6dS8Vzk8+2XhkvHJl43xKR9KlQrOYKFatajxCLX6GDt2wYCwdTZeqMYpVAf37Nx7JFv/2\n6bNhMih3VyxW7wwa1BjTxhs31qs3rY5Zt65xnVasaKzeKf0tfUrzLX2GDXvzuq5fnz59/WztXqnq\njehZ8abBjYB/B0Z0ZKHWfa1dm+rAn366sRBfsCAVeE2PhFetenP97siRqRpk883T0fNDD8HDD6f5\nrluXpunTJxVqpfEGDWqsxli3DsaNS0fFO+6YhpX06dNYeG688YYJY/DglqsuequGhvLOlsxaUlEV\nlqS7I2LXKsRTEZ+BVM8TT8Cf/gR//jM88AC8+GJqUBwzJhXiEyfCllumo9tS4V08+i1eYbJ+PSxd\nms4qFi1KBf7226fPppumcUrVSMW6dDPrfDU5A5G0S6GzgXRG4pPebqhU7168pDAiFealM4HHH09J\nYtmydKbxyivw4Q/Dl74Ekye/uQG1s0kp6ZhZ11dOG8jMQuda0lN5z4yIx6oZWHv4DKTRU0/BDTfA\njTems4dSfXmpiqlfvzdf9TJqVDoL2G472GqrVM00fHg609h6a1dzmPVENX0fSFfWmxPI2rVw662p\nmukvf0k3PO23H+y/P2y77YaNxa1dR29mvUutrsL6AfCjiFieu4cDX4+I73ZkwZ2pNySQiHQj1+OP\nw6OPNlY53XVXutzyoINSVdMuu/iMwczaVqsEcm9E7Nyk3z0RsUtL09RaT0sg69bB/PnpzOLWW+GO\nO1LiGDw4NVpvs01jldMuu6QrlczM2qNWl/H2kTQgIl7PCx0IDGhjGivTyy/DlVfCJZekJLFsWWq4\nHj0a9twT9torNWBvvfWGl6yamdVbOQnkEuBGSRfk7qOAC6sXUs91yy0wc2bjPRLPPAPXXw/77puS\nxE47NbZXuBrKzLq6shrRJR0I7J87r4+Iv1U1qnbq6lVYN98Mp5ySnjM0dWo6kyg9YO0jH0lXPZmZ\n1VKt2kAmAIsjYlXuHgiMiogFHVlwZ+pqCWTNGpg9O11Oe8018PzzcPLJcPjh6TJaM7N6q1UCuQt4\nT0Sszt39gdsiYreOLLgz1TuBRKQro264IVVJ3XJLujJq//3T533vc+Iws66lVo3ofUvJAyAiVuck\nYqSb9Y4+Ol01dcAB6SzjvPMaH81hZtZTldNU+7ykg0sdkg4BXqheSN3D+vXw85/D7rvDwQenRPKr\nX8GnPuXkYWa9QzlVWJNIV2KNBgQ8BXw2Iv5V/fDKU+sqrJtvhu98J1VdnX9+ui/DzKw7qemjTCQN\nAoiIVySNioglHVlwZ6pVArn5Zpg+PT1v6uST4dOf9qNBzKx7qlUbSHHcj0s6DNiWdEbSKyxeDF/5\nCtx9N0yblto5/BIeM+vtWm0DkTRQ0lRJVwP3A2cCpwJjaxFcva1fD+eem15ctM026flTRxzh5GFm\nBq2cgUiaAewB/A04C7gZ+FdEzKpNaPWzciXMmAFnn53eE33TTfCOd9Q7KjOzrqW1M5DtgOeAR4BH\nImIdzbwbvac54wwYPx6uugrOPBP+8Q8nDzOz5rR4BhIRO0l6O3AoMFPS88DgrtaA3pl+8xu48EK4\n7TZ429vqHY2ZWdfWnquwdiUlk08CT0fEe8qc7kBSFVgDcF5EnNFk+DDgfGASsBI4OiIeLmfawjw6\nfBXWQw/BlCkwa1Z6VLqZWU9WlzcSShKwd0TcUsa4DcBcYD9gETAbmBoRjxbG+RGwIiJOlbQNcE5E\n7F/OtIV5dCiBvPoq7LYbfPObcOSRFc/GzKzb6IwE0u6HhkfSZvLIJgPzImJhRKwBLgMOaTLOdsBN\ned6PAVtK2rTMaTvFscfC5MlOHmZm7VHtt06MId25XvJ07lc0B/gYgKTJwDjSZcLlTNthl12W3vh3\nzjmdPWczs56tK9zR8EPgbEn3AA8A9wLr2juT6dOnv/F9ypQpTJkypc1pliyB44+HP/85Xa5rZtZT\nzZo1i1mzZnXqPMt5FtYA4OPAlhQSTkR8r82ZS3sA0yPiwNx9Ypq0+cbwPM584J3AO8qdtpI2kAj4\n+MfTDYKnn96uSc3Mur1aPcrkj8BLwN3A6+2c/2xgK0njgcXAVNKVXG+QNBR4LSLWSDoGuDk/b6vN\naTvid7+Dxx6DSy/trDmamfUu5SSQsaWzgPaKiHWSjgOuo/FS3EckfSENjnNJz9W6UNJ64CHgc61N\nW0kcTRWrrgYM6Iw5mpn1PuVUYZ0L/HdEPFCbkNqvvVVYn/hEulHQVVdm1lvVqgprL+DI3DbxOumd\nIBERO3RkwfVy5ZXppsGLL653JGZm3Vs5ZyDjm+sfEQurElEFyj0DefHF9FyrK66A95R1H72ZWc9U\nszvRJe0I7J07/x4Rczqy0M5WbgI54ggYNiw9ZdfMrDerSRWWpOOBY4A/5F4XSzo3Iv67IwuutWuv\nhVtugQe6bEuOmVn3Uk4V1v3AuyPi1dy9CfDPrtQG0tYZyJo1sPXW6eVQ739/DQMzM+uiavUsLLHh\nneHrcr9u48or0zs+nDzMzDpPOVdhXQDcIemq3P1R4LzqhdS5ItKLoU4+ud6RmJn1LOU2ou9CupwX\nUiP6vVWNqp1aq8L6+9/hc5+DRx+Fhmo/OtLMrJuo6lVYkoZExMuSRjQ3PCJe7MiCO1NrCeSjH4UP\nfAC++MUaB2Vm1oVVO4H8OSI+km8gLI5UupFwYkcW3JlaSiDz5sGee8KCBbDxxrWPy8ysq6rLGwm7\nopYSyLHHwogRcOqpdQjKzKwLq0kCkXRjROzXVr96ai6BvPgibLUVPPwwvPWtdQrMzKyLquqNhJI2\nAjYG3iJpOI2X7g6hCm8G7GzXXw977+3kYWZWLa1dxvsF4KvAaNK7QEoJ5GXg51WOq8PuvBP22KPe\nUZiZ9VzlVGF9uas/tqS5Kqy99oJTToH9ukxFm5lZ11HLhym+A9gO2KjULyJ+25EFd6amCWTNGhg+\nHJ55BoYOrWNgZmZdVK0epjgNmEJKINcAHwRuBbpMAmnqwQdh3DgnDzOzairn3uxPAPsBz0bEUcCO\nQJcumu+8E3bfvd5RmJn1bOUkkJURsR5YK2kI8BywRXXD6pg77oDJk+sdhZlZz1ZOArlL0jDgV6Sr\nse4B/lnVqDrIZyBmZtXXrjvRJW0JDImI+6sVUCWKjegrVqR7P5Yvh3796hyYmVkXVe0bCXdpbVhE\n3NORBVfLXXfBTjs5eZiZVVtrV2Gdmf9uBLwLmEO6mXAH4C7g3dUNrTJ33un2DzOzWmixDSQi9o2I\nfYHFwC4R8a6I2BXYGXimVgG21x13uP3DzKwWymlE3yYiHih1RMSDwLbVC6ljfAZiZlYb5bzS9n5J\nvwYuzt2HA12qEb3kmWfg9ddhwoR6R2Jm1vOVk0COAr4IHJ+7bwH+X9Ui6oDS2Yc6dF2BmZmVo80E\nEhGrgJ/lT5c2ezbstlu9ozAz6x1au4z39xHxSUkPsOErbQGIiB2qGlkF5s+Hj3yk3lGYmfUOrZ2B\nlKqsuk2RvHgxbL55vaMwM+sdWkwgEbE4/11Yu3A6xgnEzKx2WqvCWkEzVVekmwkjIoZULaoKOYGY\nmdVOa2cgg2sZSEe98kp6kZTfAWJmVhvlXMYLgKTN2PCNhE9WJaIKLV4Mo0f7El4zs1pp8050SQdL\nmgfMB24GFgDXVjmudnP1lZlZbZXzKJNTgT2AuRExgfR2wturGlUFnEDMzGqrnASyJiKWAg2SGiJi\nJunpvF2KE4iZWW2V0wayXNIg0iNMLpH0HPBqdcNqPycQM7PaKucM5BBgJfA14K/A48BB1QyqEk4g\nZma11dp9IOcAMyLitkLvC6sfUmWcQMzMaqu1M5C5wE8kLZD0I0k71yqoSjiBmJnVliKau9m8MII0\nHpiaPwOBS4FLI2Ju9cMrj6QYMSJ49FHYdNN6R2Nm1vVJIiI6dOdcmwmkyQJ3Bs4HdoiIPh1ZcGeS\nFP36BatWQUM5rTpmZr1cZySQcm4k7CvpIEmXkG4gfAz4WEcWWg1vfauTh5lZLbXWiP5+4FDgQ8Cd\nwGXAf0SpjcBoAAAJmUlEQVREl7uEF9z+YWZWa63dB3ISMAP4ekQsq1E8FXMCMTOrrdaexvu+WgbS\nUU4gZma1VfVWA0kHSnpU0lxJ32pm+EhJ10q6T9IDko4sDFsgaY6keyXd2dpynEDMzGqr7Me5V0JS\nA/Bz0gMYFwGzJf0xIh4tjHYccF9EfFDSW4DHJF0cEWuB9cCUcqrQnEDMzGqr2mcgk4F5EbEwItaQ\nGuIPaTLOs0Dp5VWDgaU5eUB6+2FZMTqBmJnVVrUTyBjgqUL307lf0a+A7SUtAuYAxxeGBXC9pNmS\njmltQU4gZma1VdUqrDKdBMyJiH0lTSIljB0i4hVgz4hYLGnT3P+RiLi1uZnMmDGdP/0pfZ8yZQpT\npkypUfhmZl3frFmzmDVrVqfOs113ord75tIewPSIODB3nwhERJxRGOca4PulhzZKuhH4VkTc1WRe\n04AVEfHTZpYTa9YEfbtCOjQz6wZqcid6B80GtpI0XlJ/0vO0rm4yziPA/gCSRgFbA09I2ji/hwRJ\nmwAHAA+2tCAnDzOz2qpqsRsR6yQdB1xHSlbnRcQjkr6QBse5wOnABZLmkBrNvxkRL0qaAFwlKXKc\nl0TEddWM18zMylfVKqxakRQ9YT3MzGqlO1RhmZlZD+UEYmZmFXECMTOzijiBmJlZRZxAzMysIk4g\nZmZWEScQMzOriBOImZlVxAnEzMwq4gRiZmYVcQIxM7OKOIGYmVlFnEDMzKwiTiBmZlYRJxAzM6uI\nE4iZmVXECcTMzCriBGJmZhVxAjEzs4o4gZiZWUWcQMzMrCJOIGZmVhEnEDMzq4gTiJmZVcQJxMzM\nKuIEYmZmFXECMTOzijiBmJlZRZxAzMysIk4gZmZWEScQMzOriBOImZlVxAnEzMwq4gRiZmYVcQIx\nM7OKOIGYmVlFnEDMzKwiTiBmZlYRJxAzM6uIE4iZmVXECcTMzCriBGJmZhVxAjEzs4o4gZiZWUWc\nQMzMrCJOIGZmVpGqJxBJB0p6VNJcSd9qZvhISddKuk/SA5KOLHdaMzOrn6omEEkNwM+BDwDbA4dK\nenuT0Y4D7ouInYB9gTMl9S1zWmti1qxZ9Q6hS/B2aORt0cjbonNV+wxkMjAvIhZGxBrgMuCQJuM8\nCwzO3wcDSyNibZnTWhP+B0m8HRp5WzTytuhc1U4gY4CnCt1P535FvwK2l7QImAMc345pzcysTrpC\nI/pJwJyIGA3sDJwjaVCdYzIzszYoIqo3c2kPYHpEHJi7TwQiIs4ojHMN8P2IuC133wh8C+jb1rSF\neVRvJczMeqiIUEem79tZgbRgNrCVpPHAYmAqcGiTcR4B9gdukzQK2Bp4AnipjGmBjm8EMzNrv6om\nkIhYJ+k44DpSddl5EfGIpC+kwXEucDpwgaQ5gIBvRsSLAM1NW814zcysfFWtwjIzs56rKzSiV6w3\n32goaaykmyQ9lG/A/EruP1zSdZIek/Q3SUPrHWutSGqQdI+kq3N3r9wWkoZKulzSI3n/2L0Xb4uT\n8ja4X9Ilkvr3lm0h6TxJSyTdX+jX4rrnbTUv7zcHlLOMbptAfKMha4H/ExHbA+8Gjs3rfyJwQ0Rs\nA9xEusqttzgeeLjQ3Vu3xdnANRGxLbAj8Ci9cFvk9tNjgJ0jYgdSlf2h9J5tcQGpfCxqdt0lbQd8\nEtgW+CDwC0ltti132wRCL7/RMCKejYj78vdXSBcjjCVtgwvzaBcCH61PhLUlaSzwIeDXhd69bltI\nGgLsHREXAETE2oh4iV64LYCXgdXAJpL6AgOBZ+gl2yIibgWWNend0rofDFyW95cFwDxSGduq7pxA\nfKNhJmlLYCfgdmBURCyBlGSAzeoXWU39DDgBKDbq9cZtMQF4QdIFuTrvXEkb0wu3RUQsA84EniQl\njpci4gZ64bYo2KyFdW9anj5DGeVpd04gBuSbLq8Ajs9nIk2viujxV0lI+jCwJJ+RtXba3eO3Bama\nZhfgnIjYBXiVVG3RG/eLicDXgPHAaNKZyOH0wm3Rig6te3dOIM8A4wrdY3O/XiOfll8BXBQRf8y9\nl+T7aZD0VuC5esVXQ3sCB0t6ArgUeJ+ki4Bne+G2eBp4KiLuyt1XkhJKb9wv3gXcFhEvRsQ64Crg\nPfTObVHS0ro/A2xRGK+s8rQ7J5A3blKU1J90o+HVdY6p1s4HHo6Iswv9rgaOzN+PAP7YdKKeJiK+\nHRHjImIiaT+4KSI+A/yJ3rctlgBPSdo699oPeIheuF8AjwF7SNooNwjvR7rIojdtC7HhWXlL6341\nMDVfpTYB2Aq4s82Zd+f7QCQdSLripHSj4Q/rHFLNSNoTuAV4gHQaGsC3ST/670lHEwuBT0bE8nrF\nWWuS9gG+HhEHSxpBL9wWknYkXUzQj/RUh6OAPvTObXECqcBcB9wLfJ701O8evy0kzQCmACOBJcA0\n4H+By2lm3SWdBHwOWEOqEr+uzWV05wRiZmb1052rsMzMrI6cQMzMrCJOIGZmVhEnEDMzq4gTiJmZ\nVcQJxMzMKuIEYr2epBX573hJzb71sgPzPqlJ962dOX+zenICMWt8HtAE4LD2TCipTxujfHuDBUXs\n1Z75m3VlTiBmjU4H9spPsT0+v6DqR5LukHSfpGMg3e0u6RZJfyQ9JgRJV0manV/u9fnc73RgYJ7f\nRbnfitLCJP04jz9H0icL855ZeCHURYXxfyjpwRzLj2q2VcxaUNV3opt1MyeSH4MCkBPG8ojYPT9v\n7TZJpcc77AxsHxFP5u6jImK5pI2A2ZKujIiTJB2bn4pbEnneHwd2iIh3StosT3NzHmcnYDvg2bzM\n95BeCvXRiHh7nn5ItTaCWbl8BmLWsgOAz0q6F7gDGAG8LQ+7s5A8AL4q6T7SO1nGFsZryZ6kJwcT\nEc8Bs4DdCvNeHOk5Q/cBWwIvASsl/VrSvwErO7huZh3mBGLWMgFfjoid82dSfiERpPdspJHSAxzf\nB+weETuRCv2NCvMod1klrxe+rwP65seRTyY9vv8jwF/bvTZmncwJxKyx8F5BelJryd+AL+X3riDp\nbfntfk0NBZZFxOv5vfR7FIatLk3fZFl/Bz6V21k2Bfamlcdn5+UOi4i/Av8H2KH81TOrDreBmDVe\nhXU/sD5XWf0mIs7Orwu+J79P4jmaf3/2X4H/lPQQ6R0U/ywMOxe4X9Ld+R0lARARV0naA5gDrAdO\niIjnJG3bQmxDgD/mNhZIb9ozqys/zt3MzCriKiwzM6uIE4iZmVXECcTMzCriBGJmZhVxAjEzs4o4\ngZiZWUWcQMzMrCJOIGZmVpH/D+/uRPNxf223AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x118412cd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_data(validation_accuracy, 'loss', [0, 100, 0.8, 1], \n",
    "          'Iterations', 'Validation Accuracy', \n",
    "          'Validation accuracy change over iterations')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, as the model trains over increasing number of iterations over the training data:\n",
    "\n",
    "    1. Cross Entropy Loss Decreases per iteration\n",
    "    \n",
    "    2. Training Accuracy keeps increasing rapidly at beginning, \n",
    "    but tends to saturate as number of iterations increase. \n",
    "    We shall allow maximum of 100 iterations to prevent overfitting.\n",
    "    \n",
    "    3. Validation accuracy also increases as the model trains over \n",
    "    repeated iterations.\n",
    "    \n",
    "Now that we have trained our model successfully, let us report the results. __The model accuracy that is to be reported is the accuracy over test data.__ \n",
    "\n",
    "But first, let us calculate the final train accuracy and validation set accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Set Accuracy:  0.92498\n"
     ]
    }
   ],
   "source": [
    "pred_output_train = np.dot(add_ones(train_dataset), weights)\n",
    "print \"Training Set Accuracy: \", accuracy(raw_train_labels, \n",
    "one_hot_encoding(pred_output_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Set Accuracy:  0.926\n"
     ]
    }
   ],
   "source": [
    "pred_output_valid = np.dot(add_ones(valid_dataset), weights)\n",
    "print \"Validation Set Accuracy: \", accuracy(raw_valid_labels, \n",
    "one_hot_encoding(pred_output_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9225\n"
     ]
    }
   ],
   "source": [
    "pred_output_test = np.dot(add_ones(test_dataset), weights)\n",
    "print accuracy(raw_test_labels, one_hot_encoding(pred_output_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The accuracy of our LOGISTIC REGRESSION classifier on test MNIST data is 0.9225 or 92.25 %.\n",
    "\n",
    "Now let us move on to train our next algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single Layer Neural Network\n",
    "\n",
    "Next we shall be using a Neural network with one single layer, for attempting to classify the handwritten digits. We shall use the same MNIST dataset, which we split into train-validation-test for training and testing purpose. \n",
    "\n",
    "Here are the Neural Network parameters used:\n",
    "\n",
    "    1. Input Layer - 784 neurons\n",
    "    \n",
    "    2. Hidden Layer - 500 neurons\n",
    "    \n",
    "    3. Output Layer - 10 neurons\n",
    "    \n",
    "    4. Batch size for Stochastic Gradient descent - 100\n",
    "    \n",
    "    5. Number of training iterations - 5\n",
    "    \n",
    "    6. Gradient Descent Learning rate - 0.5\n",
    "    \n",
    "We shall initialise the weights between the input layer and hidden layer; and hidden layer and output layer randomly(normal) using a standard deviation of 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Validation accuracy:  0.9526\n",
      "1\n",
      "Validation accuracy:  0.961\n",
      "2\n",
      "Validation accuracy:  0.9693\n",
      "3\n",
      "Validation accuracy:  0.9704\n",
      "4\n",
      "Validation accuracy:  0.9738\n"
     ]
    }
   ],
   "source": [
    "train_size=50000\n",
    "test_size=10000\n",
    "features=784\n",
    "k=10\n",
    "hidden_layer = 500\n",
    "neural_neta = 0.5\n",
    "neural_b_size = 100\n",
    "neural_b_start = 1\n",
    "neural_iterations = 5\n",
    "hidden_wts = np.random.normal(0, 0.2, (hidden_layer, features))\n",
    "neural_out_wts = np.random.normal(0, 0.2,(k, hidden_layer))\n",
    "hidden_wts = hidden_wts/len(hidden_wts[1])\n",
    "neural_out_wts = neural_out_wts/len(neural_out_wts[1])\n",
    "n = np.zeros(neural_b_size)\n",
    "targetValues= train_labels.T\n",
    "validation_accuracy = []\n",
    "train_losses = []\n",
    "num_iter=0\n",
    "while(num_iter < neural_iterations):\n",
    "    neural_b_stop = min(train_size,neural_b_start+neural_b_size-1);\n",
    "    curr_train_design_mat = train_dataset[\n",
    "        neural_b_start:neural_b_stop]\n",
    "    curr_train_output_k_format = train_labels[\n",
    "        neural_b_start:neural_b_stop]\n",
    "    curr_train_size = len(curr_train_design_mat)\n",
    "    for j in range(0,neural_b_size-1):\n",
    "        train_line = curr_train_design_mat[j]\n",
    "        hidden_inp = np.dot(hidden_wts,train_line)\n",
    "        hidden_out = 1/(1 + np.exp(-1 * hidden_inp));\n",
    "        second_inp = np.dot(neural_out_wts,hidden_out)\n",
    "        neural_out_line = 1/(1 + np.exp(-1 * second_inp))\n",
    "\n",
    "        train_out_line = curr_train_output_k_format[j]\n",
    "        \n",
    "        second_err = np.multiply(\n",
    "            np.multiply(neural_out_line,(1 - neural_out_line)),\n",
    "                                 (neural_out_line - train_out_line))\n",
    "        hidden_err = np.multiply(\n",
    "            np.multiply(hidden_out,(1 - hidden_out)),\n",
    "            np.dot(neural_out_wts.T,second_err))\n",
    "        neural_out_wts = neural_out_wts - \n",
    "            neural_neta * np.dot(np.vstack(second_err),\n",
    "                                 np.vstack(hidden_out).T)\n",
    "        hidden_wts = hidden_wts - \n",
    "            neural_neta * np.dot(np.vstack(hidden_err),\n",
    "                                 np.vstack(train_line).T)\n",
    "    train_losses.append(np.sum(neural_out_line - train_out_line))    \n",
    "    neural_b_start = neural_b_start+neural_b_size\n",
    "    if(neural_b_start>train_size):\n",
    "        print num_iter\n",
    "        trained_val=np.zeros(len(valid_dataset))\n",
    "        test_input = valid_dataset\n",
    "        test_pred = []\n",
    "        for i in range(len(test_input)):\n",
    "            test_inp_line = test_input[i]\n",
    "            hidden_inp = np.dot(hidden_wts, test_inp_line)\n",
    "            hidden_out = 1/(1 + np.exp(-1 * hidden_inp))\n",
    "            second_inp = np.dot(neural_out_wts,hidden_out)\n",
    "            neural_out_line = 1/(1 + np.exp(-1 * second_inp))\n",
    "            test_pred.append(neural_out_line)\n",
    "        print \"Validation accuracy: \",accuracy(\n",
    "            raw_valid_labels, one_hot_encoding(np.array(test_pred)))\n",
    "        validation_accuracy.append(\n",
    "            accuracy(raw_valid_labels, one_hot_encoding(np.array(test_pred))))\n",
    "        neural_b_start = 1\n",
    "        num_iter = num_iter+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEZCAYAAACNebLAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XeYVOXZx/HvDYiFomLBQpFii7FrJKJSVOwtJrEnxsQS\nfS3RGIzGQKKxJRo10SRq1MSaxIZdY5DEWEBFwIYiKhCaIIigCCzc7x/3GXd29szszO7Ozpbf57rm\nmplTnzkz89znKec55u6IiIjkalfpBIiISPOkACEiIqkUIEREJJUChIiIpFKAEBGRVAoQIiKSSgGi\nAsysnZktNrMejbmstC5mNsPM9mriff7MzG5spG3VK/1mtreZfdAYaZCGUYAoQpJBf5o8VprZ51nT\njil1e+6+yt27uPv/GnPZUpnZJWZ2a2NvV5o3MzvCzCaY2Sdm9pGZ/TNzAuLul7r76ZVOI1DxC7TM\nrL2ZrTKz8TnTLzezm5LX/ZJlHspZ5h4zu7Ap01sOChBFSDLoru7eFZgGHJQ17Z7c5c2sfdOnUhpL\na/7+zGwL4FbgTHdfB+gD/AFYVdGEVVgd33lPM/tmHZvY3cx2acw0NQcKEKWz5FE9Ic7E7zWzu81s\nEXCcmQ0wsxfNbKGZzTSz6zI/wqwzk17J+zuS+Y8npZLnzax3qcsm8w8ws3eS/V5vZv81s++U/CHN\nvmJmY5LtTDSzA7PmHWxmbyX7n25mZyfTNzCzx5J1PjazMVnrbGpmDyRnrFPN7PSsebuZ2atmtsjM\nZpvZlQXSdZqZTTGzecn2uifTbzKzy3OWfdTM/q+I/df6/lL2u7qZXZN83tlm9nsz65jM29vMPjCz\ni81sfrL9o7LWXdvM7kz2/b6ZXZCz7VPN7O3keE4ys22zZu+cTFtoZneZ2Wp1Hes67AhMcffnANz9\nM3d/wN1nZR2LW5PXmbPjEyyqi+aa2fCsdK+ZfK6FZvaGmQ23PFVDFi40s/eS43C3ma1dTILN7KLk\nmH5qZq+b2SHJ9NWTfW+ZtexGZvaZma2bvD/UorS00Mz+Y2bbZC07w8x+bGaTgCUFknAV8EszswLL\n/Bq4rJjP06K4ux4lPIAPgKE50y4BvgAOTN6vDuwM7EoEk82AycDpyfz2wEqgV/L+DuAj4s/bHrgX\n+Gs9lt0Q+BQ4OJn3I2AZ8J08n+US4NaU6asB7wPnJdvZG1gM9E3mfwTslrxeB9gheX0VcD1x4tEB\n2COZbsBrwPBke32T7Q9J5o8DjkpedwJ2zZPeYcAcYFugI3AD8K9k3hDg/axluwGfAesXsf9a31/K\nvn8H3A90BToDjwK/SObtDawArkiO3ZBk35njdTdwH7AWccY+BTghmXcMUSrNHMP+wKbJ6xnAC8AG\nwLrAO8BJhY51Mu+PwLV5jmF/YCnwG2AwsFa+3wTQjyhZ3Jh8rh2T49Qvmf8b4BmgC7Ap8HrOdzAD\n2Ct5fR7wHLBR8t3dRPK7TUnj3jnb+SawYfL6aOK3uEHWZ70ka9lzgfuT17sCs4Gdkt/AicB7QIes\n9L0MbJznO8/893oD40n+R8DlwE1Zx2glsEayr8znvQe4sNL5VYPzu0onoKU9yB8gnqljvfOAvyWv\n2yd/vOxM/8asZQ8BJtVj2e8B/87Z7yxKDxCDgRk50/6e+cED/wNOAjrnLPMrIiPsmzN9d+C9nGk/\nA/6UvP5v8r5bHcfwduDSrPddgCpgkyQDmAEMSOadBjyZvB5Yx/4Lfn/JtpcCPbOm7Qm8m7zem8g4\n18iafz8RkDoQwaNf1rzTgaeT188AP8yz3xnAt7LeXw1cX+hYF/kbHgD8jQj0nwN/zqSd2gFiJUlm\nnEx7FfhG8noaMDhr3qnkDxDvAntmzesJLM2TvhoBImX+68ABWd9t9j5fAw5LXt8EXJyz7nvA17PS\nd1yB/Xz53yP+Z1OTabUCRPL6TOA/yetWESBUxdR4ZmS/MbMtkyqO2Um1xS+Is9l85mS9/pw4Sy11\n2U1y00Fk5qXaBJieM20acZYIcARwGDDdzEab2deS6Zcn6/0rqQb6cTK9N9DbzBYkj4XA+UD3ZP73\ngG2Ad8zsJTM7oEC6pmXeuPtiYCFxxu1EEMt0GjgWuCt53auO/UPt45ZtI6JUODGzDeARan6fH7v7\nF1nvpyXp3ZA4y5+eMy9zLHsSGU8+c7NeZ3/XV5B+rOvk7i+5+1HuviEwCBgK/LTA8vPypGFjav6+\nCh3DXsAjWcdvErDKzDasK71mdmJSTZT57rYkOfbu/jxQZWYDk+qjnsDjyaq9geE53/tGVB97KPL/\n4e6PEAH15AKL/QnoZWb7F7PNlkABovHk9rr4E3Gm09fd1wZGkNN2UQaziT9Itk3TFqzDrJTt9AJm\nArj7y+5+GFH18RhRzYW7L3H3c929D3A48efck8g43nX3bsljXXdf290PT9ab4u7HuPsGwDXA/Zn6\n/ZR0Zbe3dCGqXmYmk+4BvmVmmxHVIQ8k0wvuP1Go18xcoqpuy6xtrOPu3bKWWc/MVs85XrOITGVV\ndrqT15k0zyDOQkvi7ovzHOtSt/My8BDw1VLXJU5Usrtf9yqw7Axg35zvoJO7f1RoB2bWh6jiOjWz\nHlHVlv1f+itwQvL4u7uvyNrnL3L22dnd78tat5TeUj9LHmukzXT35cAvgUtL2GazpgBRPl2ARe6+\n1My2Jorf5fYosKOZHWTRuH0OhUstAB2Sxr7MoyNR711lZueaWQczGwocAPzNzNYws2PMrIu7ryQa\n91bCl43XfZPtLiaqf1YBLwLLk+2tnqTtq2a2U7Le8Wa2XrLep8k6ab1q7gG+n6y7OlFi+Y8nDazu\n/kqy35uAx939s2S9gvuvi7uvAm4BrjOz9ZM09zCzfbMWaw+MNLPVzGwwsD/wD3evAv4BXGZmnZIM\n7xyiqpBkuz8xsx2S7fY3szqDeoFjXdd6e5nZ981sg+T91kT1yYv5Vimwub8DF1o0wvcgqs7y+RNw\nuZn1TPa7YaaxuQ6dic81P/neTga2ylnmTqKd4hgiWGTcDJxhSe8iM+ucHLc1i9hvLe7+L6Kq7ISc\nWdnH6C9EO9W+tAIKEKUr9ozjPOBEM/uU6EZ4b4Ht1LXNopZNzsaOAn4LzCcaRF8jzn7zOY6oNvic\nqGefnJwJHUqcmc4HrgWOcfdMVch3gQ/N7BOieijT62dLYLSZLSYaJK919+eTQHIg8DXgQ+Ks+o9E\nECWZ93ZSFXcV8O0kY839fE8RZ2gPEWfgPajd4+geog77rqz16tp/Mc4jqobGJZ/7SaLBN2MG0TA9\nG7gN+L67v5/MO4Noh/gQeBa4zd3vSNJ2L3AlEXwXEW0X62aSXiA9qccawMxuNrPr86y3kKgifD35\nbT5K/DavybN8bhqy348gjuWHxPH4GzV/a9nLXgM8QVSJLSLanersFururxMdBF4mSmSbAy/lLPMh\n8AawzN1fypo+Fvgh8IekWmsyNX8vxfyXc5e5iPh+Uv+TyW9tRMoyLZIlDSrSCplZO+JPdWQm85DG\nZ2Z7Aze7e986F27FLLoUH+buTX72bGa3E43Vv2zqfbdmKkG0Mma2X1LkXx34ObCc6EYq0qjMbBMz\n+7qFrYlu1Q/UtV4Z0tGX6DShUQEamQJE67MH0cd/LlEPenhWo51IY1qdqOf/FHiKaGu5qSkTYGaX\nEdWov/IyDEfT1qmKSUREUqkEISIiqTpUOgHFMDMVc0RE6sHd6339VYspQVT6kvPm8hgxYkTF09Bc\nHjoWOhY6FoUfDdViAoSIiDQtBQgREUmlANHCDB48uNJJaDZ0LKrpWFTTsWg8LaKbq5l5S0iniEhz\nYmZ4AxqpW0QvJhGRXJttthnTpk2re8E2oHfv3nz44YeNvl2VIESkRUrOjiudjGYh37FoaAlCbRAi\nIpJKAUJERFIpQIiISCoFCBGRMujTpw+jR4+udDIaRAFCRERSKUCIiEgqBQgRkTJavnw555xzDptu\nuik9evTgRz/6EStWxD28Pv74Yw455BDWXXdd1ltvPQYNGvTleldeeSU9evSga9eubL311jz77LNN\nnnZdKCciUkaXXnop48aNY9KkSQAceuihXHrppfziF7/g6quvpmfPnnz88ce4Oy+99BIA7777Ljfc\ncAOvvvoq3bt3Z/r06axcubLJ064ShIi0WmYNfzTU3XffzYgRI1hvvfVYb731GDFiBHfccQcAq622\nGrNnz+aDDz6gffv2DBw4EID27duzfPly3njjDaqqqujVqxd9+vRpeGJKpAAhIq2We8Mf9ZW5unnW\nrFn06tXry+m9e/dm1qxZAJx//vn069ePYcOG0b9/f6688koA+vXrx7XXXsvIkSPp3r07xx57LLNn\nz27QsaiPZhEgzKydmY03s4crnRYRkcZiZmy66aY1xoyaNm0am2yyCQCdO3fmN7/5DVOnTuXhhx/m\nmmuu+bKt4eijj+a55577ct0LLrigydPfLAIEcDbwVqUTISLSWDJjIx199NFceumlzJ8/n/nz53PJ\nJZdwwgknAPDYY48xdepUALp06UKHDh1o164d7777Ls8++yzLly+nY8eOrLnmmrRr1/TZdcUDhJn1\nAA4Ebql0WkREGoslDRgXX3wxO++8M9tttx3bb789u+yyCxdddBEAU6ZMYZ999qFLly4MHDiQM844\ng0GDBrFs2TIuuOACNthgAzbZZBPmzZvH5Zdf3vSfodKjIZrZP4BfAWsD57n7oSnLaDRXEalBo7lW\nK9dorhXt5mpmBwFz3X2CmQ0G8n6QkSNHfvl68ODBumuUiEiOMWPGMGbMmEbbXkVLEGZ2GXA8UAWs\nCXQBHnD37+QspxKEiNSgEkS1cpUgKl7FlGFmg1AVk4gUSQGimm4YJCIiTarZlCAKUQlCRHKpBFFN\nJQgREWlSGqxPRFqk3r17f3mtQVvXu3fvsmxXVUwiIq2UqphERKQsFCBERCSVAoSIiKRSgBARkVQK\nECIikkoBQkREUilAiIhIKgUIERFJpQAhIiKpFCBERCSVAoSIiKRSgBARkVQKECIikkoBQkREUilA\niIhIKgUIERFJpQAhIiKpFCBERCSVAoSIiKRSgBARkVQKECIikkoBQkREUilAiIhIKgUIERFJpQAh\nIiKpFCBERCSVAoSIiKRSgBARkVQKECIikkoBQkREUilAiIhIKgUIERFJVdEAYWY9zGy0mb1pZq+b\n2VmVTI+IiFQzd6/czs02AjZy9wlm1hl4FTjM3SfnLOeVTKeISEtkZri71Xf9ipYg3H2Ou09IXi8B\n3gY2rc+2Xn0Vxo9vzNSJiLRtHSqdgAwz2wzYARhbn/X//ndo3x522qkxUyUi0nY1iwCRVC/dB5yd\nlCRKtnhxBAgREWkcFQ8QZtaBCA53uPuofMuNHDnyy9eDBw9m8ODBNeYvWQJW75o2EZGWb8yYMYwZ\nM6bRtlfRRmoAM/srMN/dzy2wTJ2N1EccEQHigQcaO4UiIi1Ti26kNrOBwHHAUDN7zczGm9n+9dnW\n4sXw6aeNmz4RkbasolVM7v480CgtB0uWgHrCiog0noq3QTSWxYth1apKp0JEpPVoNQFiyRKoqqp0\nKkREWo9WEyAWL1aAEBFpTK0iQLhHgFi5MqqZ2mkIQhGRBmsVWeny5REUOnWCzz6rdGpERFqHVhEg\nFi+Gzp2hSxd1dRURaSytoopp8eIIDmusEa9FRKThWkWAWLKkOkCoBCEi0jhaRYDIVDGttZZKECIi\njaXONggz+4aZdUleX2BmfzezHcqftOJlShBqgxARaTzFNFKPdPfFZrY7cCBwF/DH8iarNJkSRNeu\nChAiIo2lmACxMnk+GPhTMiT36uVLUukyjdRdu6qKSUSksRTTBjHbzG4A9gd2MbOONLPusapiEhFp\nfMVk9N8G/g0c5O4LgfWBC8qaqhJlVzGpBCEi0jiKCRDrA6PcfbKZ7QEcDjxf3mSVpjWXIP74xxhC\nRESkqRUTIB4CVplZP+A2YHPg7rKmqkSttQSxciWccQbMmlXplIhIW1RMgFjl7iuAbwC/c/cfAZuW\nN1mlyTRSt7YSxPz5Mfjg3LmVTomItEXFBIgqM/sWcALwaDJttfIlqXSZKqbW1s01ExgUIESkEooJ\nECcBQ4Cr3P19M+sD3FPeZJUme7C+1lTF1NYDxPLlMG9epVMh0nbVGSDc/Q3gLOAVM9sKmOHuvyp7\nykrQWksQc+bEc1sNEHfeCcceW+lUiLRddV4HYWZ7AncAMwEDNjKzE9y92fRkaq0Xys2dC6ut1nYD\nxIsvwtix0Vjfvn2lUyPS9hRTxfRb4EB3H+juuwMHAdeVN1mlaa33g5g7F77ylbYbIMaOhWXL4K23\nKp0SkbapmADR0d2//Iu6+9tAx/IlqXSZKqZOnWDp0tZz3cDcubDddm0zQCxZAlOnwhFHwEsvVTo1\nIm1TMQFivJn90cz2SB5/AF4rd8JKkSlBZG47umRJpVPUOObMabsB4tVXYdttYa+9FCBEKqWYAHEa\n8D7wk+TxPnBKORNViqoqWLEC1lwz3remdoi5c2H77dtmgBg7FnbbDb7+dQUIkUopphfTF+5+lbsf\nmjx+DdzaBGkrypIlUXowi/etqR1i7lzYeuv4PCtWVDo16ZYtA/f6r//JJ7DDDrWD+rhxESC23Ram\nTYvlpG4LFkTjvkhjqO+orHs2aioaIFO9lNEUXV2XLGlYpliMlSvh44+he3dYb73mez3A0KENy5D+\n+U+YOBEeeqjm9EwJokMH2HnnCBhSt1tugSOPbL4nFNKyNKthu+sj00Cd0RRVTEccAU89Vd59zJ8P\n66wT3Vw33LB5VjN98UVk3O+8U/9tPP447LEH3HVX9bRZs6KzQd++8X7AAFUzFevZZ+P3/+CDlU6J\ntAZ5A4SZbZfnsT3NaKiN3BJEU1QxTZoEb79d3n3MnRulB4jn5hggJkyINqBp0+q3/qpV8MQTcOON\nEQAyn3HsWPja16qrDdUOUZwVK+D55+HKK+OYijRUoQvlbigw773GTkh9ZS6Syyh3CWLBAvjoI3iv\nzEegJQSIcePi2H/4Yf3Wf+21KCVtuy0ccgj87W9w1lnV7Q8Zu+0G3/9+VOtlgobU9uqrsNlmcPLJ\ncOml8MYb8NWvVs+fPBnWXz8eIsXIW4Jw9z0LPZoykYXkVjGVuwTxzjuRSZU7QMyZAxttFK9LDRCZ\nEWCnTIHx4+GDD8qTxnHj4LDD6h8gHn8cDjwwXh93XHU1U6b9IWPjjeN7fffdBiW3Yj76CG67rfz7\nGTMGhgyJaslTTqlZipg8GQYOhH33bT2dOKT8WnwbRFojdTlLEJMnR/VHcy5BDB8OW24J++8fGe+h\nh5YnjWPHwre+1bAAcdBB8XqffWI777wDr7wCu+5ac9mWWs30zjuR9tNPj8BdTs8+C4MHx+tTToF7\n741gMG9eHOerr472nG98IwZCFKlLiw8QTV2CmDw5Mt7//a+8f7L6Bgj3qKp58cW4Enns2ChBNHav\nqwULIk3DhsHs2dEWUYr582MIjT32iPcdOsDRR8PFF1f33MqW3VD91lvVy37xRcM/S0OtXBnB7oQT\n4Je/rB4a5L//hUGD4MIL48QlM/hiOaxYEd/5oEHxfpNNIujefHOU8o45Bk48EX7/+/iPfO975Q9Y\n0vK1+ADR1CWIt9+Oq5s33bT+jbPFmDu3flVMEydCx46w1VbxvmvXqHJYsKDweu+9F2eZxe7n5Zej\n++kaa0Qvq5kzi1sv46mnoovs6qtXTzvuOPjHP2pWL2UMGBBnyKedFmfJO+0UGfGOO0bDbGOaOzfq\n8Os6ZsuWwSWXQJ8+MHJklCwXLID99ovjf8QR8Je/RPtJnz7lq+qD+D769YNu3aqnnX46nH8+9OoV\ngQti0MO7747f7s9+Vr70SOtQZ4DI05Opt5k1i+CS1khd7hLE1ltD//7lrWaaM6d+JYhHHokG3+zG\n3N696w5mV14ZmfygQcVl9uPGRYYI0TCaW820bFmcOecruWS3P2Tsumsc18x2s+2wQ/VFke+8Az/5\nCdx/f2Tk3/oW/PrXdae5WBdeCPfdF5n8736X/5qCc86B556DUaPieJx5Jlx7LUyfDn/9a5zR77df\nLFtsgHCP7/7FF6MkOH9+cWkeM6a6eilj0KBoh7j99hiGJmPNNeO6k9tvj/SL5OXuBR/Ay8AKYAIw\nEVievJ4K7F3X+o3xiGSm+9GP3K++uvr9Qw+5H3JI3sUb5Isv3Fdf3X3ZMvcf/tD9+uvLsx939+22\ncx8/Pl7PmuW+4YbFrbfrru7/+lfNaYcc4v7gg/nXmTXLfZ113OfNc7/iCvd+/dw//LDwfg4+2P2+\n++L1cce5/+UvNee/8oo7VH+GbFVV7uut5z5jRu15L7/sPn9+4X2nrbPddqWtk89rr7l37+7+ySfu\nr7/uvs8+7lttFZ8n2+23u2+xhfuiRcVt96c/df/lLwsvs2iRe7du7uuv7/61r7nvt5/7Zpu5T5xY\nc7lHH410ZX9H++zjPmpUcWnJePBB9/793T/7rLT1SvXvf7uPHl3efTQXy5a5P/+8+wsvNGwbRx4Z\n/8NLLkn/nxQryTvrnfcWUwr4ENjZ3Xdw9+2BnYF3gf2AqxsaoMxsfzObbGbvmtnwUtcvpopp4MDo\n8tdQU6fG2XjHjuUvQWS3QWywQVRd1DVK7Zw50XNpz5w+ZnWVIK67Do4/Pro/Dh8eZ8KDBsWV3Gnc\n6y5BTJoUpZh77629/rhxUUXXo0ftebvsUrv9oS5bbRXfRVqd+sKF0UaSJrf9wh3OPRdGjIC1144u\nok8/HdVH++8fVylDXP/x4x/DAw/E760YxZQgpkyJYzJvXrQdPfkk/OpXsPfecca/YEG0c5x5ZgwD\nP2xYLLtsWbTP5H7vdTn88Ci1lbOqaeHCaC86+ui4ALKSSq0Grctrr8E998Bvfxsl2n32id/uGWdE\nu0+hqs+VK6PkecklcYwyVqyIY1VVFTfMmjkzqrSPOCI9/QsXRi+5cikmQGzt7pMyb9z9deAr7t7g\n7DGppvo9EWy2AY5J7lpXtLoaqauqon62MYZqmDy5um6/nAEiM8zGBhvE+w4d4nqBuqobHnssMo3V\nci5jLBQgFi2KhszzzquedvbZMUjgP/+Zvs706VGXncng8wWI446LAJFbzXTbbfDNbxb+LKXo3DmO\nT9of6NxzI329ekVV1I9+FNU+m2wS651zTnWgePjhCMwnn1y9vhkcdVRUxVxzTTTuHnkkXH89bLNN\n8WksJkBMnRrtCNmOPTaq4848M3qmdesWx/a66+Db34YDDoDRo2GLLWDddYtPT8b110cmVygzK7UD\nQrbhwyNz++EP4aSTyj9ETT7z5sVv4Lvfjd98Qz32WJw0jBoV/611143/zfTpEThuuy0y+tyq4YUL\n4aqr4nu+/PLIQ7beOpZfsSJOAJYti+rFAQPgD3+IDjE77BBtfqNGxXaWL4/qzC23jPWvuKJMnWbq\nKmIA9wG/AwYmj+uTaasDrzSk+AIMAJ7Ien8BMDxlubxFqIMOcn/44er377zjvvnmNd+D+znnFF0q\ny+vSS91/8pN4/eabUcVQDnPmRDVDtm22qV3VkOuww9zvuKP29L//3f0b30hf58or3Y89tvb0X//a\n/cwz09f5299iXxn//Kf7kCE1lxkyxP2JJ9y33rpmcXvuXPd113X/6KPCn6VUgwa5P/NM7em77BL7\nnzLF/c474/M+8oj7tGlRlXXkke7bb+8+YUL8bp54Iv8+Fi92P/746t9AKd57z71378LLXH65+49/\nnD5vzhz3SZNqTlu1yv2009w7dXI/99zS05Rx//3x2XO/k1Wrogp3zTXdzz7b/X//K227Y8a49+gR\nVWfLl8d3ccMN9U9nQzz8sPsee7ifckp8D//+d+HlP//c/U9/ct922zjGy5ZVz3vrLfcNNnB/8cXC\n27j44vgfrFgRx/LOO6P68oQT3MeNq17u5Zfdd9st5g0b5r50afr2nn/evU+f+A1usYX7/vtHWt57\nL/LBLbeM/2I2GljFVEwmvhYwHHgkeVwAdALaA2s3aOdwJHBT1vvjgetTlsv7Jey1l/uzz1a/nzUr\nDnTGgw/GH2jo0LybKNrxx7vfemu8Xro02iNWrGj4dnNNnBgBIdvQoe5PP51/naVL3bt2Ta+/HzvW\nfeeda0//4gv3jTeOzDHX88+777RT+r7OO8/9V7+qfv/uu+59+1a/X7Uq2hhmzXL/xS8ic8kYOTL+\npI3t5JPd//CHmtNWrozvvlA7wapV7rfc4t65c/zhymX5cveOHeM5nx/8wP3GG0vbblVVBPKxYxuW\nvosvrtnesXKl+xlnRJvWm29GAFp33cgsc+vEly1z//nP42Tg2mujTWPp0sjEHnqoernJk+N38c47\nDUvro4/GCcHAgZGx7ref+8yZhde58ML4jO5xgrDxxrHuD34QbYkPPOD+17/G8R8+PNr8Dj7Y/ckn\n43nw4PhvffxxtNvcfnvd6ayqct93X/dTT42Mf/vt839PK1e6P/ZY3e1Bn3zifsEFsWyuhx9279XL\n/fTTq7dT9gBRzkcpAWLEiBFfPp7Nigg77VSzAXHJEve11qp+f9llcYa8/vqRGTRE5mw0o2dP9/ff\nb9g20zz9tPvee9ecdswx6aWDjMcec99zz/R5aSUSd/fbbos/V5ovvojjuHhx7Xl77lnzTOWLLyLz\nq6qK9zNnVh/vyZPjz1hVFZlG9+5x1tPYfv3rONvN9sEHcQZbjA8+iNJNOfXq5T51av75Q4a4P/VU\nedNQyD33xPd2330RxL/+9ciQMj76KEpP3bpFJrpgQZRqdtzR/cADI+2HHx7f8X77Reks1+9+F5lt\nfU2fHmfv997r/txzcRY/YkSUgAqVcIYMcX/88er3ixfH+jfeGEHvsMOis8Vpp7lfdFHN32hVlfv5\n50ej8V57lVZamzcvTs6uvLLwyUFjeeSRZ33bbUf4euuN8O98Z0STlCAGAE8AbxGN0+8C7zZkpznb\nfjLrfclVTJtvHplQxqpV7u3aVZ/ZH3+8+5//HGcus2eXerhrbrdz5ziDyBg8uPBZfX399a+1q33O\nOcf9N7+40ZHoAAAV7UlEQVTJv85pp8WPMM2qVe5rrBHBM9t3vhNnz/nsvnvtHlErVsRZ+cKFNadv\nskn8ed2jmia7xLbDDlHKu+WWyEjKYdSoKGZne/TROHNrLvJVg2X07h3VBZX0yivum24a1TGffpq+\nzIwZcea9/vrx+POfa558vf66+1lnRQky19KlUWX1+eelp62qKv5zl11We94VV8SZfVqPn6oq9y5d\nav536+O229xPPLE8tQaN7d57o2diQwNEMY3UtwE3AvsQ94HIPBrDy0D/5LqKjsDRwMOlbCC3kdos\n3md6Mr39djTibLttNO7V16xZsNZaNS9E6t8/GhYzVq6M/u/ewIa47B5MGYWuhZg3L4Z3zjekhhn0\n7BkNaNkmTozG6Hx23x1eeKHmtBdeiAa2ddapOT27oXrSpOh5kXHUUdEQes010WhcDptvXnusprfe\nKq0hudwKNVQvXx69rXr1ato05dp55zhuzzxT83+VrUeP6Njw4ovRo+ukk2ped/PVr0Yj+sYb1153\njTWio8fEibXnvfhijO6bz9VXR0+1n/yk9rzhw2N4kcGDa/9P3ngjOiVk/3fr48QTozG5Q6EhTpuJ\no45qnF5jxQSIT939EXef5e5zM4+G7xrcfSXwf8DTwJvAve5e0kDaud1cobqr66pV1Re2bbcdvP56\n/dOa2U623J5MTz8dvSQaOmZQKQFixYrozXLSSdU9rNLk9mRavjwy1EIZ6MCBtXu33HVX9KwptP3c\nwHPUUdV/rKFD8++vIfr2jQCYfVHbW29Fd9DmolCAmDYtuv7m9kCrhK5da17hnk///pHmUu28c4w8\nm+uWW+IkIs348fCb38QJWPv26cucf34MVnjTTTWnv/RSjIfV1mRuw9wQxQSI0WZ2uZntmn01dcN3\nHdz9SXff0t03d/crSll31Sr4/PPaASLT1XXGjPixZ4aUbkgJ4u23a2fAuQHij3+MjPHmm+u/H6g5\nzEZGvgBx3nlRsrnkksLbzA0Qb78dZ/2FfkSZAfIy1xcsWxZXGKcFiEIliD59ojRywQXlG6579dUj\ns8rubvvmmy0nQKR1cW2tdtklBmTM9d//xgnJsmW15511Vlwt37t34W2fdFKcxGSX4l96KbqMSumK\nCRB7JI9riHtE3EBcu1Bxn38eGVy7nE+RKUG8/XZ1BtEYJYhCAWLGjOgr/8ADUd2TO9zHc89Fv+Vi\nZA+zkZEWIG67LS6muuuu/GdVGb1716xiqqt6KbPP9devHnzu8cfjOPbsWXvZTIBYtiyOSW7G/Mwz\nMWBcOW2xRXU1k3vN7785qCtAZO6g19qllSDmzYvf/VZbxUWCufNef72438+AAVGKHD++etqLL7bN\nEkRjqDNAePr9IPZqisTVJa16CapLEG+9VV0ttM02kcnX96KftADRrx+8/36cYd9yS1wY1rdvXPl6\n993Vyy1fHhdfXXZZcSNoFlPFNHFi1LuOGlW7PSBNbgli4sS4+KYu2e0Qd94ZV1ynyQSIyZMjI1xj\njZrzm6LeNrsdIrv02FwUChDvv992ShDbbhsnEZ9/Xj3thRcic99nnxiUMduTT8Z/qmPHurdtFiXc\nO++M9wsWRF18c2qLakkK3XL0mOT5rLRH0yUxv9wG6oy0EkSnTlEFMWVKcdv++OO4QnXmzDgbTQsQ\nnTvHkAzTpkWAOPXUmH7yyTWrmX73uwgcG25Y3BXdaVVMG24YZ1KrVkVj+CmnxJWYue0i+aQFiLpK\nEFDdDrFwYZQCjjyy8PZzq5eaUnYJorlVL0E02i5aVDNjzGhLVUyrrx7fTXZD9fPPx29t6NDaAeKx\nx6rvG1KMzBX8K1dGaWTXXesuYUu6QiWIzIX7G+R5VFzuSK4ZmRJEpgdTxnbb1W6HyDe+0c9+Fpnw\nTjtFwFm0KL2HSf/+UXXUt2/17R333TcCzPjxUWy+/PIYr+WQQ+DRRwt/ptxhNjI6dozPtWBBBJ/V\nVothH4rVq1d1gHAvPkBkShD33RfDeOQ7I+/VK87aJ0wobrvlkB0gmlsPJoiq0F690m+w1JaqmCCq\nmbLbITIBYo89YvrSpTG9qio6fxxwQPHb3mqr6LU0erTaHxqq0C1Hb0yeL057NF0S88tXxZQZ8ju7\niglqN1Q//3wU+3MH91u8OMZC+c9/4mx+2rSoA81t64AIEDfeGPcpyGjXLu4BcPPNMXT0SSfFmCkH\nHxzDcRcyf35kwmlVMt27Rzp+/vMYoyUtPfn06BHBasWKKHKb1S6lpPnKVyJNmQH98llzzRiP5skn\nK1uCyJQQm1sPpoy0aib3qGJqSwFil12q2yG++CJOLHbbLf7P221XXa35wgtRfbnJJqVt//jjo21O\n7Q8NU8z9INY3s5+Y2Y1mdlPm0RSJq0uhKqb33otMcMMNq6dnN1SvWhWDa7VrB3/6U83177orustl\nfpTdusWPNE3//lHNlFv18r3vRT3oU09Vj5Y5YEBkzrnXI2RLa3/I6N49qrG++90IdqVYbbVYf+bM\n6tJDMT2K2rWLdM+eXfdZ3GabRcZcqQDRs2dUw33+efOsYoL0ADF3bgTYtdeuTJoqIbsE8eqrcdaf\nOdkbOjTO/qHmbWlLcfTR0T6Xe39zKU0xTYejgJeA/wJ1DDjdtAo1Uj/5ZJQesjPB7BLEX/4S1TYP\nPRQ3rvm//4uGVfforlrsDWj23z/qlnMbZXv0iP7/++5bPSR0+/aRyT76aNztK+Oii6KetUOHyNzS\nhsGGyODffz+Go66PTDtBsQ3UGQccENU1dTUS9u4dbTX50l9u7dvHWfiUKS2rBNGWGqgzvvrV+Nyf\nfVZdvZQxZEjcThbif1GfbuMbbxyllA8/rF1dK8UrJkB0cvfz6l6s6eVrg+jaNTLBE0+sOb1v36gu\nmTkzMuVRo6qH0b311si0x46NH+3eexeXhp13jkeazP0Dsh1ySHRPzQSI0aPhjjvi7mhmUeeaL4M9\n5phYLy0oFiM7QBx8cPHrnXlmcctttlmUHsp1rUMxttgiGjk7dWr4lbPl0KdP7Qsp21IDdUbHjnHS\nMWFCBIjsa2t23z1O5N58M6pFd921fvs49dSoYpL6KyZAPGFmw9z96bKnpkT5qpi6dIm69twePu3b\nx4/y+OPjzD7zw7vooiiSnnxylB5OPbW0+v1SDBsW7ROffRb7OPVUuOGG4v4Ehx/esH1nAsSECfGZ\ni1Vshr/77qXf7KexbbFFlAqbY+kB0ksQba2BOiNzwdwLL8R/IGPNNWPehRdGCb2+PZC++c3Gve9I\nW1RMNnga8KSZLTGzBWa20MzquJ170yjUSA3pmcS220ZX08svr542YECcwf3+95G55JY8GtPaa8ed\n2J55Jm4kv+OOUapoCr16Rc+u6dMLD8tRX4ceGsMdVNLmm8dFic2tB1OGqpiq7bxzjNHVqVPtUvPQ\noXEDp/q0P0jjKSZArA+sBqxNdG9dn2bSzbVQCQLSrxH41rciEOT2irjooriN5MEHx9XD5XTwwTGu\nzJ//HHf0aiq9e0ej+ZZbNo8xf8phiy2iA0JzLUF06xbpy77NZFusYoIoJYwdW7P9IWPIkChhDxvW\n9OmSanmrmMxsc3efQtwKNE0DRjZqHIsXp/cu6to1zkrShoTYb7/0bQ0ZEsHjnHMaNYmpDjkkbn15\n883FdTVtLL17x3UUTVViqYQttojn5hogzKI6aeLEGHkU2m4V0zbbxEVzaQHi61+P0nxzbEdqSwq1\nQVwAfJ8YeymXAxUfbiNfI3X//tHTp5TGUrO4+rIp9OsXvaya+uwoM9BZKT2YWpru3eP7z1y02BwN\nHx5X+44eHVUrixaV3s+/NVhttWj7S/sfdOjQuk9kWoq8AcLdv588N9a9HxpdoesgKl0XXpd8JZly\n6tQpGpErdaVzUzArfjiVSjnmmLhSeJ99YhiWPn3K1ymiubv99kqnQAopagg1M9sK+ArwZW9/d787\n/xpNI18jteR3ww26srQ5OOmk6NJ89NERKESaozoDhJn9DBgGbAU8BexHXDRX8QCRrwQh+R11VKVT\nIBmnnBIlh0peNyJSSDEliKOAHYDx7n6CmW0M3F7WVKWYOjWGzV68OAaFmz49uguqBCEt2Q9+UOkU\niORXTIBY6u4rzazKzLoAc4A67uvU+IYNi6sv11or+vP37BkX0mR6rYiISOMqJkC8ZmbrALcCrwCf\nAkXc1aBxTZ3a1HsUEWnbzLNv3po708yAjdx9dvK+P9DV3cfnXakMzMwLpVNERGozM9y93q1cBQNE\nsoM33L2ivcoVIEREStfQAFFM7+sJZrZjfXcgIiItU94ShJl1cPcqM3sT2BKYCnwGGODuvlOTJVIl\nCBGRkjW0BFGokXocsBNwaH03LiIiLVehAGEA7q7+QyIibVChALGBmZ2bb6a7X1OG9IiISDNRKEC0\nBzqTlCRERKRtKdRIPb4pG6ILUSO1iEjpytnNVSUHEZE2rFAJopu7N4t7T6sEISJSurJfSd0cKECI\niJSuKa6kFhGRNkgBQkREUilAiIhIKgUIERFJVbEAYWZXmdnbZjbBzO43s66VSouIiNRWyRLE08A2\n7r4DMAX4aQXTIiIiOSoWINz9GXdflbx9CehRqbSIiEhtzaUN4iTgiUonQkREqhUarK/BzOyfQPfs\nSYADF7n7I8kyFwEr3P3ucqZFRERKU9YA4e77FppvZicCBwJD69rWyJEjv3w9ePBgBg8e3LDEiYi0\nMmPGjGHMmDGNtr2KDbVhZvsDVwN7ufvHdSyroTZERErUYsdiMrMpQEcgExxecvfT8yyrACEiUqIW\nGyBKoQAhIlI6DdYnIiJloQAhIiKpFCBERCSVAoSIiKRSgBARkVQKECIikkoBQkREUilAiIhIKgUI\nERFJpQAhIiKpFCBERCSVAoSIiKRSgBARkVQKECIikkoBQkREUilAiIhIKgUIERFJpQAhIiKpFCBE\nRCSVAoSIiKRSgBARkVQKECIikkoBQkREUilAiIhIKgUIERFJpQAhIiKpFCBERCSVAoSIiKRSgBAR\nkVQKECIikkoBQkREUilAiIhIKgUIERFJpQAhIiKpFCBERCSVAoSIiKSqeIAws/PMbJWZdat0WkRE\npFpFA4SZ9QD2BaZVMh0iIlJbpUsQvwXOr3AaREQkRcUChJkdCsxw99crlQYREcmvQzk3bmb/BLpn\nTwIc+BlwIVG9lD0vr5EjR375evDgwQwePLixkiki0iqMGTOGMWPGNNr2zN0bbWNF79Tsq8AzwOdE\nYOgBzAS+5u4fpSzvlUiniEhLZma4e8GT74LrN4eM18w+AHZy94V55itAiIiUqKEBotKN1BlOHVVM\nIiLStJpFCaIuKkGIiJSutZQgRESkmVGAEBGRVAoQIiKSSgFCRERSKUCIiEgqBQgREUmlACEiIqkU\nIEREJJUChIiIpFKAEBGRVAoQIiKSSgFCRERSKUCIiEgqBQgREUmlANHCNObtBFs6HYtqOhbVdCwa\njwJEC6MffzUdi2o6FtV0LBqPAoSIiKRSgBARkVQt5pajlU6DiEhL1JBbjraIACEiIk1PVUwiIpJK\nAUJERFI16wBhZvub2WQze9fMhlc6PU3JzHqY2Wgze9PMXjezs5Lp65rZ02b2jpk9ZWZrVzqtTcXM\n2pnZeDN7OHnfJo+Fma1tZv8ws7eT38dubfhY/DQ5BpPM7C4z69hWjoWZ/dnM5prZpKxpeT97cqym\nJL+bYcXso9kGCDNrB/we2A/YBjjGzLaqbKqaVBVwrrtvA3wdOCP5/BcAz7j7lsBo4KcVTGNTOxt4\nK+t9Wz0W1wGPu/vWwPbAZNrgsTCz3sDJwI7uvh3QATiGtnMsbiPyx2ypn93MvgJ8G9gaOAC40czq\nbLxutgEC+Bowxd2nufsK4F7gsAqnqcm4+xx3n5C8XgK8DfQgjsFfksX+AhxemRQ2LTPrARwI3JI1\nuc0dCzPrCuzp7rcBuHuVuy+iDR4L4FNgOdDJzDoAawIzaSPHwt3/CyzMmZzvsx8K3Jv8Xj4EphB5\nbEHNOUBsCszIev+/ZFqbY2abATsALwHd3X0uRBABNqxcyprUb4Hzgexud23xWPQB5pvZbUl1201m\nthZt8Fi4+0LgamA6ERgWufsztMFjkWXDPJ89Nz+dSRH5aXMOEAKYWWfgPuDspCSR2y+51fdTNrOD\ngLlJiapQsbjVHwuiGmUn4AZ33wn4jKhWaIu/i77Aj4DewCZESeI42uCxKKBBn705B4iZQK+s9z2S\naW1GUmy+D7jD3Uclk+eaWfdk/kbAR5VKXxMaCBxqZu8D9wBDzewOYE4bPBb/A2a4+yvJ+/uJgNEW\nfxe7AM+7+wJ3Xwk8COxO2zwWGfk++0ygZ9ZyReWnzTlAvAz0N7PeZtYROBp4uMJpamq3Am+5+3VZ\n0x4GTkxefxcYlbtSa+PuF7p7L3fvS/wORrv7CcAjtL1jMReYYWZbJJP2Bt6kDf4ugHeAAWa2RtLg\nujfRiaEtHQujZqk632d/GDg66eXVB+gPjKtz4835Smoz25/osdEO+LO7X1HhJDUZMxsI/Ad4nSgm\nOnAh8aX+nTgbmAZ8290/qVQ6m5qZDQLOc/dDzawbbfBYmNn2RGP9asD7wPeA9rTNY3E+kSGuBF4D\nfgB0oQ0cCzO7GxgMrAfMBUYADwH/IOWzm9lPge8DK4gq66fr3EdzDhAiIlI5zbmKSUREKkgBQkRE\nUilAiIhIKgUIERFJpQAhIiKpFCBERCSVAoS0CWa2OHnubWbHNPK2f5rz/r+NuX2RSlGAkLYic8FP\nH+DYUlY0s/Z1LHJhjR2571HK9kWaKwUIaWsuB/ZIRkI9O7kJ0VVmNtbMJpjZyRBXbJvZf8xsFDGU\nBWb2oJm9nNzA6QfJtMuBNZPt3ZFMW5zZmZn9Oll+opl9O2vbz2bd9OeOrOWvMLM3krRc1WRHRSRF\nh0onQKSJXUAyVAdAEhA+cffdkjG/njezzBAEOwLbuPv05P333P0TM1sDeNnM7nf3n5rZGcnIqhme\nbPtIYDt339bMNkzW+XeyzA7AV4A5yT53J278c7i7b5Ws37VcB0GkGCpBSFs3DPiOmb0GjAW6AZsn\n88ZlBQeAc8xsAnFfjh5Zy+UzkBh9Fnf/CBgD7Jq17dkeY91MADYDFgFLzewWMzsCWNrAzybSIAoQ\n0tYZcKa775g8+iU3nYG410IsFIMEDgV2c/cdiEx9jaxtFLuvjGVZr1cCHZIhq79GDPF+MPBkyZ9G\npBEpQEhbkcmcFxOjfWY8BZye3HsDM9s8uUNbrrWBhe6+LLk3+ICsecsz6+fs6zngqKSdYwNgTwoM\nsZzsdx13fxI4F9iu+I8n0vjUBiFtRaYX0yRgVVKldLu7X5fc0nV8ck+Bj0i/h/GTwGlm9iZxH4IX\ns+bdBEwys1eT+1Q4gLs/aGYDgInAKuB8d//IzLbOk7auwKikjQPibmkiFaPhvkVEJJWqmEREJJUC\nhIiIpFKAEBGRVAoQIiKSSgFCRERSKUCIiEgqBQgREUmlACEiIqn+H6UhEIzHbIsAAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116927910>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_data(train_losses, 'loss', [0, 100, -5, 5], \n",
    "          'Iterations', 'Training Loss', \n",
    "          'Training Losses over epochs: Single layer NN')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we are using batch gradient descent, there are a lot of fluctuations in the training losses, which increase every time model encounters a new batch of data, but over many epochs the effective losses decrease and go down to zero as expected.\n",
    "\n",
    "Let us visualise, the increase in validation accuracy over 5 iterations of the model over train data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEZCAYAAACAZ8KHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYXVWZ7/Hvr0hCgJCBKQwZIcwIBCEmCBoUlaEFlUFA\nQJAGr4pgi42odBPlNgLt3EJ3I4Mgk4gi2A3KFYggEKYMQAhhCkmAEAiEGZJQee8fa59kV+VU1dmV\n2nVOpX6f56mn9rzfs87w7rXWHhQRmJmZ1aqp3gGYmVnP4sRhZmaFOHGYmVkhThxmZlaIE4eZmRXi\nxGFmZoU4cXSSpJGSlktqysZvlnRMLct2Yl/fkXTR6sRr9SHpPyV9r84xPCrpI/WMIa+r4pH0UUnz\nO7nuZZJ+sLox9Fa9NnFIukXSpCrTD5a0oMYf+RUXwUTEARHxm1qW7SCuVb4MEfHDiDiplvWtsUTE\nVyLi32D1fuhqVe0HMSJ2iog7y9xvqxj6SvqxpPmS3pD0jKSflBRPj74QTdIXs4PKb7WaPr+SXCVN\nypY5NDd/rWzaiO6OGXpx4gAuB46uMv1o4DcRsbyb46kQPfzLUCtJa9U7hq5Uw+tZrfe2B5XXd4Hd\ngN0jYiAwEZha14gaQDvv36vA6ZLWa2N+AK8A35ekVtProjcnjj8CG0raqzJB0mDgH4ArsvEDJE2V\n9LqkuZLOamtjku6Q9KVsuEnSjyS9LOkp4MBWyx4n6bHsaOwpSSdl09cFbgY2l/RmNn9TSWdJ+k1u\n/YOy6v6rkm6XtF1u3hxJp0maIWmxpGsk9Wsj5i0l3SZpkaSXJF0paWBu/jBJv8/mvSzpF7l5J+Ze\nw6OSds2mL5e0ZW65FUfAlSNuSadLWgBcKmmwpD9l+3glG948t/4QSZdKej6b/4ds+iOSDswt1yeL\ncZc2XuuJkp7MXusfJW2aTb9Q0r+3WvaPkr6RDW8m6fosvqclfT233FmSfifpN5JeA75YZb+XSfpB\nO++tJJ2RfQ5elnRt9jnMN3F+SdJc4LZs+nVKteLFkiZL2r7yGoEvkH6E3pB0Y+4z8bFsuJ+kn2Xl\n+Zykn0rq2+r9+aakhdkyx+VeywGSZmbbni/pm9XKGtgduCEiFgJExLyIuDK3nXw8Z0n6raTLs+0+\nImm33LK7aeV38LqsfKo2MbX3XrWnvc+gpEMlPdhq+W9KuiFXnj9S+n1YkH2e1m5Vnis+722EMAu4\nFzitnTD/AiwF8s3hamPZ0vXaxBER7wG/A47NTf48MCsiHs3G3wKOiYhBpB///yPpoBo2fxJwALAL\n6Ut0aKv5C4EDsqOx44GfSto1It4B9gdeiIj1I2JgRLxYCRlA0jbA1cApwMbALcCfJPXJbf8w4JPA\n6CyG49qIU8A5wKbA9sAwYFK2nybgf4A5wAhgC+DabN5hwL8CR2ev4SDSEdGKONuxKTA42+ZJpM/g\npcDwbNo7wAW55a8E1sni2wT4aTb9Clp+iQ4klduMVV5k+pE6h/Q+bAbMA36bzb4GODy37GBS2V0j\nScCfgGnZeh8HTpX0idzmDwKui4jBwFVtveh23ttTsm3sDWwOLAYubLX6R4DtgE9l4zcDW2XlMZX0\neSAifpXFcH62/YOrhHImMA7YmfTZGJdNq9gUWD+L5R+BCyQNyuZdDJyYvec7Abe38XKnAKdJ+oqk\nndoqk5xPZ69hEKm8L4DU5AX8gfT52ID0Xn222gZqfK/a0t5n8CZglKRtc8sfTWqxADgPGEMqzzGk\n78m/5pZt/XmvJoB/Ab5ROWioYnm2zFlqhJpnRPTaP+DDpC9qv2z878Cp7Sz/U+DH2fBIoBloysbv\nAL6UDd8GnJRb7xP5Zats9wbg69nwR4F5reafBVyRDZ8JXJubJ+A54CPZ+BzgyNz884ALayyPg4GH\nsuEJpAS3SszAnyvxVpm3HNgyN34Z8IPca3sP6NtODLsCr2TDmwHvAwOrLLcZ8CYwIBv/HfCtNrZ5\nMXBubnw90tHbiGz8WWCvbPgfgb9mwx8Cnm21rTOAS3Lvy+QOyrT162/93j4G7NPqdS0l/ZhVPmMj\n29n+4KzM12+9v9wyc4CPZcNPAZ/Kzfsk8Ewuvrfz73n2GRiXK6cTK/tqJyYBXwHuAt7NPp/HthHP\nWcCtuXnbA29nwx8B5rfa9l3VyrOj96q996W9z2A2fiFwdja8I+kgqU82/hYwOrfshFbl2dHn/YvA\nndnwb4EfZsPzWfmdzn//pwBfBtbK3vcRtXy3u/qv19Y4ACLibuBl4DNKzSt7kB29AUgap9QU9FLW\nFPFlYKMaNr056Y2vmJufKWl/Sfdm1eLFpCPRWrZb2faK7UX6NM0nHelULMwNvwMMqLYhSZsoNWU9\nl72+K3NxDAPmRvW+nuHA0zXG29rLEbEsF8M6kv5b0rNZDH8DBmdHkMOAVyPijdYbiYgFpER/SHZE\nvD9tH/G3LrO3SV/+Spn9FjgyGz4qt50RwBZKTYKvZu/Vd0hH+hWr29k9Erihsg9SIlkGDM0t81xl\nQKkZ9Nysaes10o9wUOzzMy83PjebVvFKq/c8//k5hFSzm6vUNDu+2g4i+c+I2JuU2M4hNUtuW215\n4MXc8DtA/6zGuxnwfKtl2yrvWt6rqjr4DEKq3R6VDR9NqmG+L2ljYF3godz7dwuwYW7zLT7vHfhX\n4CuS2ov5TOB7QP8at1mKXp04Mr8hZf2jgb9ExMu5eVeT+kK2iNQU8d/U1q64gPTjWjGyMqDU33A9\ncD6wcUQMIX3YKtvtqKnnhfz2MsPJ/bgUcA7pqGXH7PUdnYtjPjBC1c8um09qKqnmHdKXqWLTVvNb\nv77TgK2BPbIYKqdpKtvPBsr1u7RSaa46DLgnSybVtCgzpU7IDVn5o3QNcKjSGSofAn6fTZ9POnrc\nIPsbEhGDIuLT7bye9lRbdh6wf6t9rNfqteTXO4rUtPOxrLxGkcqqs5+fkdm0joOPeCgiPkNqIr0R\nuK6GdZZExIWkmv0OtewnZwEtD4ig5fcqr5b3qi3fou3PIBExBVgqaW9S+Vf6GxeRPu875vY7OFLT\ndkXNn4+ImE1qmvteW+tFxF9JtcavFtl2V3PiSD8++5KaKC5vNW8AsDgilkkax8qjjoq2ksh1wCmS\ntpA0BPh2bl6/7G9RRCyXtD+puaBiIanTvq0fy+uAAyXto9Qh/C1Sdfje9l9mVeuTqtpvStoC+Ofc\nvPtJX9xzJa0raW1Je2bzLga+VenElLSVpMoXehpwVHZkvB+put5RDO8Cb0jagKyPBSBSH8AtwIVZ\nB2af7MtbcQPp7J1TyE5oaMM1wPGSds46Ls8BpkTEvGw/00k1kIuBP+dqOPdnZXO6pP5Kp0DuKGn3\nDl5TW6q9t/8NnJMlLSRt3KofrfVnbH1gCbA4S4A/pOUPyEJgS9p2DXCmpI0kbURqN2/vNHKyuPpK\nOkrSwIhoJjUTNrex7KlZx3ClzL5I+i7VemZV5TXfCzRL+lq2nYNJfTLVrM57NYA2PoM5VwK/BJZG\nxD2worb/K+BnWe2D7Dv/ySrr1+oHpH7Ptvo6INU6Tl+Nfay2Xp84ImIucA/pKPmmVrO/Cpwt6XXS\nm/Xb1qu3Mfwr0lkQM4AHWXkES0S8Rfqh+11WtT2CdPRWmT+b9OV+Jqv+tjhij4gnSDWDX5Ka2Q4E\nPh0R71eJoyPfBz4IvEbqWMzHuZx0ZLs16ah4PlknckRcD/wbcLWkN0g/4Btkq36D1Nm7mNT8c0MH\nMfyMVPaLSO/Dza3mH0Pq53ic9KN4ai7G90hHaKOz/1VFxG2kH8g/kGoZo0nlnnc1qUP1qtx6y0ln\n2e1KahJ6ifTetpXUq+4+t71q7+3PSe//rdnn7B5a/ji2fj+vIL0fzwOPZsvnXQLsmG2/Uib5bfxf\n0mfyYVZ+Pv+tlvhJ78WcrDnnJFY9kKp4B/gx6cDjZVJ/x+ey71q111R1n1kTz+dIB3WLs/39iZQ4\nW66weu9VR59BSMl1J1ZNst8m1QCmZOVyK7BNDfusKiKezfbR1qm5ZInrfupY41DW4VLOxqVLSG/m\nwojYuY1lfkFqn34bOC47+iM7Wv0ZKbldEhHnlRao9ViSzgS2iYhjO1zYejxJU4D/jIjWrQNl77c/\n6cBlt4jobP/eGqPsGsdlrDyFcBVZM81WEbE1qeP5v7LpTaQj6k+RzmI4UrlrFcwAsmaFEwDfjmUN\nJekjkobmmrw+QDqrr7t9FXjASSPp0/EinRcRf5fUuiM372CytumIuE/SIElDSU0JT1aqtpKuzZZ9\nvMx4reeQ9I+kGunlEfH3esdjpdmW1K+3LvAMcEhkFxZ2F0lzssHPdOd+G1mpiaMGW9Dy9LrnsmnV\nprfVKWa9UERcTOrMtjVYpIsaf1XnGEbXc/+NqNE6x+t2Cb2ZmdWm3jWO52l5XvawbFo/0gU9radX\nJaluZxeYmfVUEdGpg/XuqHHkL05q7Saye0VlV6G+lrVfPgCMUbrJWz/SqZOtT5VtoR6X3Rf5O+us\ns+oeg+N0nI7TcVb+VkepNQ5JV5NuqbyhpHmke670I107c1FE3Kx0x82nSKfjHk+a2SzpZNI50ZXT\ncWeVGauZmdWm7LOq2rpAKL/MyW1M/zPpjAozM2sgjdY5vsaaOHFivUOoiePsWo6zaznOxlDqlePd\nRVKsCa/DzKy7SCI62Tle77OqzGwNNmrUKObOndvxglaakSNH8uyzz3bpNl3jMLPSZEe19Q6jV2vr\nPVidGof7OMzMrBAnDjMzK8SJw8zMCnHiMDOzQpw4zMysECcOM7M66MlnmzlxmFmvdd555zFmzBgG\nDhzITjvtxB//+McV8371q1+xww47rJg3ffp0AJ577jkOOeQQNtlkEzbeeGNOOeUUAL7//e9zzDHH\nrFh/7ty5NDU1sXz5cgD22WcfzjzzTPbaay/WW2895syZw69//esV+xgzZgwXXdTyYZY33ngjY8eO\nZdCgQWy99dbceuutXH/99ey+++4tlvvJT37CZz/72VLKqKp636Gxi+7yGGbWeBr9u3n99dfHiy++\nGBER1113XQwYMCBefPHFuO6662LYsGHx0EMPRUTE008/HfPmzYvm5ubYZZdd4rTTTot33303lixZ\nEnfffXdEREyaNCmOOeaYFdt+9tlno6mpKZqbmyMiYuLEiTFy5MiYNWtWNDc3x7Jly+Lmm2+OOXPm\nRETEnXfeGeuuu25MmzYtIiLuu+++GDRoUNx2220REfHCCy/E7NmzY8mSJbHhhhvG448/vmJfY8eO\njRtuuKHqa2zrPcimd+o31zUOM6sbqWv+OuuQQw5h6NChABx22GGMGTOG++67j0suuYTTTz+d3Xbb\nDYAtt9yS4cOHc//997NgwQLOP/98+vfvT79+/dhzzz1r3t9xxx3HdtttR1NTE3369GH//fdn1KhR\nAOy999588pOf5K677gLg0ksv5YQTTuBjH/sYAJttthnbbLMN/fr14/DDD+fKK68EYObMmcydO5cD\nDzyw8wVRkBOHmdVNRNf8ddYVV1zB2LFjGTJkCEOGDGHmzJksWrSI+fPns9VWW62y/Pz58xk5ciRN\nTZ376Rw+fHiL8VtuuYUJEyaw4YYbMmTIEG655RYWLVq0Yl/VYgA49thjufrqqwG48sorOfzww+nb\nt2+nYuoMJw4z65XmzZvHSSedxIUXXsjixYtZvHgxO+64IwAjRozg6aefXmWd4cOHM2/evBX9Fnnr\nrbce77zzzorxBQsWrLKMctWjpUuXcuihh3L66afz8ssvs3jxYvbff/8VnebDhw+vGgPA+PHj6dev\nH3fddRdXX311i76V7uDEYWa90ttvv01TUxMbbbQRy5cv57LLLuPRRx8F4IQTTuBHP/oRU6dOBeDp\np59m/vz5jBs3js0224wzzjiDd955hyVLlnDPPfcAsOuuu3LnnXcyf/58Xn/9dc4999x297906VKW\nLl3KRhttRFNTE7fccgu33nrrivknnHACl112GXfccQcRwQsvvMDs2bNXzD/66KM5+eSTCzeXdQUn\nDjPrlbbffntOO+00xo8fz6abbsrMmTPZa6+9ADj00EP53ve+x1FHHcXAgQP57Gc/y6uvvkpTUxN/\n+tOfePLJJxkxYgTDhw/nuuuuA2Dffffl85//PDvvvDN77LEHn/70p1vsT606YwYMGMAvfvELDjvs\nMDbYYAOuvfZaDj744BXz99hjDy677DK+8Y1vMGjQICZOnMi8efNWzD/mmGN49NFHu722Ab47rpmV\nyHfHLc97773H0KFDmTp1apt9IeC745qZWebCCy9kjz32aDdplMUPcjIz62FGjx4N0OKCxe7kpioz\nK42bqurPTVVmZlZ3ThxmZlaIE4eZmRXiznEzK83IkSNXuX7ButfIkSO7fJvuHDcz64XcOW5mZt3G\nicPMzApx4jAzs0KcOMzMrBAnDjMzK8SJw8zMCnHiMDOzQpw4zMysECcOMzMrxInDzMwKKT1xSNpP\n0uOSnpD07SrzB0v6g6QZkqZI2iE37zuSZkp6WNJVkvqVHa+ZmbWv1MQhqQn4JfApYEfgSEnbtVrs\nu8C0iNgF+CLwi2zdkcCJwNiI2Jl0Q8YjyozXzMw6VnaNYxzwZETMjYhlwLXAwa2W2QG4HSAiZgOj\nJG0MvAEsBdaT1AdYF3ih5HjNzKwDZSeOLYD5ufHnsml5M4DPAUgaB4wAhkXEYuDHwDzgeeC1iPhr\nyfGamVkHGuF5HOcCP5c0FXgEmAY0S9oS+CdgJPA6cL2koyLi6mobmTRp0orhiRMnMnHixJLDNjPr\nOSZPnszkyZO7ZFulPo9D0nhgUkTsl42fAUREnNfOOs8AOwMHAJ+IiBOz6ccAH4qIk6us4+dxmJkV\n0MjP43gAGCNpZHZG1BHATfkFJA2S1DcbPhG4MyLeAmYD4yX1V3qE2MeBWSXHa2ZmHSi1qSoimiWd\nDNxKSlKXRMQsSV9Os+MiYHvgcknLgZnACdm6MyRdATwENJOasC4qM14zM+uYHx1rZtYLNXJTlZmZ\nrWGcOMzMrBAnDjMzK8SJw8zMCnHiMDOzQpw4zMysECcOMzMrxInDzMwKaYSbHJqZWYmWL4dXX4WF\nC1f+rQ4nDjOzHqiSDF58sWVCWLhw1WkvvwwDBsDQoSv/VodvOWJm1iCWL4dXXmk7AeSnLVoE66/f\nMhkMHQqbbrrqtE02gbXXbrmv1bnliBOHmVmJli9PP/Lt1Qgq0xYtgoED204A+WmbbAL9+nU+LicO\nJw4z60bNzalmUEsz0aJFMGhQx7WCTTeFjTdevWRQRKmJQ9JaEdHcqci6iROHma2u5uaWNYP2ksIr\nr6Rk0FGtoFIz6Nu33q9uVWUnjmeA3wOXRcRjndlJ2Zw4zKya5ubUMdxRraCSDAYP7rhWMHRoqhk0\nYjIoouzEsT7pyX3Hk677uBS4NiLe6MwOy+DEYdZ7vP/+yppBR01Fr74KQ4bU1oG8JiSDIrqtj0PS\nR4GrgcHA9cDZEfFUZ3bclZw4zLpfczMsWQLvvdc9/999NyWMxYtXJoOOmoo23hj6+KKDqlYncXRY\npJLWAg4k1ThGAT8GrgL2Bm4GtunMjs2sc5qbu/ZHubPrNjenUzz791+9/xtuWPvyG22U/pwM6quW\n4n8SuAP494i4Jzf9ekkfKScss57t1VdhyhR47rmuP/pevnz1f6z7908XhOXHi26jb19Qp45Xraer\npY9jQES81U3xdIqbqqyempvh0UdTorj33vS3YAHssQeMHt25H+X2/vfp4x9sW31ld45fDpwaEa9l\n40OAH0fElzqzwzI4cVh3WrSoZZJ48EHYfHMYPx4mTEh/O+4Ia61V70jN2lZ24pgWEWM7mlZPThxW\nlvffh0ceaZkoXnoJxo1LCWL8ePjQh1I7vVlPUmrnONAkaUhELM52tkGN65n1OC+91DJJPPQQDB+e\nEsTee8Ppp8P227s2Yb1bLQngx8C9kn4HCDgU+LdSozLrBsuWwcMPt0wUr7ySahATJsAZZ6ThIUPq\nHalZY6npOg5JOwL7ZKO3N9oV5G6qslosXLgyQdx7L0ydCqNGteyb2G47aPLjzawX6JYLACVtAvSv\njEfEvM7ssAxOHNba0qUwY0bL2sRrr61MEpW+iUGD6h2pWX2U3Tl+EKm5anPgJWAkMCsiduzMDsvg\nxGELFrSsTUyfDltu2bI2sc02rk2YVZSdOGYAHwP+GhFjJe0DHB0RJ3Rmh2Vw4uhdli6FadNa1ibe\neqtlbWLcuPRcAzOrruzE8WBE7J4lkLERsVzSjIjYpTM7LIMTx5rt+edXJogpU1JtYuutW9Ymtt7a\nF8WZFVH26bivSRoA3AlcJekl4O3O7MysI0uWpE7rfG3ivfdWJomzz05XZK+/fr0jNeu9aqlxrAe8\nS7ql+heAQcBVEfFK+eHVxjWOniki3cspX5t4+GHYdtuWtYmttnJtwqyrldZUld0Z968RsU+bCzUA\nJ46e4b330gV1+drEsmUrE8T48ak2sd569Y7UbM1Xdh/HbcDnIuL1zuygOzhxNJ4ImDevZW3i0UfT\nVdf52sTo0a5NmNVD2YnjRmAs8P/I9W1ExCmd2WEZnDjq7913083+8rWJiJa1id13h3XXrXekZgbl\nJ44vVpseEZd3ZodlcOLoXhHw7LMtaxOPPZbuCFtJEhMmwMiRrk2YNapue3Rsp3Yg7Qf8jNS5fklE\nnNdq/mDSc8y3InXCf6lySxNJg4CLgZ2A5dm8+6rsw4mjRG+/3bI2MWVKupCuUpuYMAF22w3WWafe\nkZpZrcquccwBVlkoIrasIbAm4Ang48ALwAPAERHxeG6Z84E3I+JsSdsCF0TEvtm8XwN/i4jLJPUB\n1o2IN6rsx4mji0TAM8+0rE08/jh84AMtaxPDh7s2YdaTlX0dx+654f7AYcAGNW5/HPBkRMwFkHQt\ncDDweG6ZHYAfAkTEbEmjJG0MLAH2jojjsnnvA6skDVs977wDDzyQksQ996RE0a/fyprE0UfD2LHp\nyXNmZlBD4qhyvcbPJD0E/GsN298CmJ8bf46UTPJmAJ8D7pY0DhgBDCM1TS2SdBmwC/Ag6UmE79aw\nX6siAubOTQmiUqOYNWtlbeKYY+CCC1JtwsysLR0mDkm75UabSDWQrnyQ07nAzyVNBR4BpgHNQF9g\nN+BrEfGgpJ8BZwBndeG+12iV6yYqtYl7703T99wzJYojjoAPftC1CTMrptYHOVW8D8wBDq9x+8+T\nahAVw7JpK0TEm8CK55dnfSrPAOsB8yPiwWzW9cC329rRpEmTVgxPnDiRiRMn1hjimqNyFXYlSTzy\nSLpuYsIEOOww+MlPfKaTWW81efJkJk+e3CXbKvWsquzK89mkzvEFwP3AkRExK7fMIOCdiFgm6UTg\nw5V+DUl/A06MiCcknUXqHF8lefTGzvHKHWIrTU733JNqGJXaxIQJ6boJX4VtZtWUfVbVOcD5EfFa\nNj4EOC0izqwxuP2An7PydNxzJX0ZiIi4SNJ44HJSn8ZM4ITKVeqSdiGdjtuXVAs5vtoV7L0hcbz4\nYsvaxPTpMGZMShCVZOF7OplZrcpOHNMiYmyraVMjYre21ulua1riqDwLO1+beP31ltdNjBvnO8Sa\nWeeVfTruWpLWjogl2c7WAdbuzM6supdfTqfBVmoTDz2U+iImTIB994V/+Rc/vc7MGkctieMq4Lbs\ntFiA40lNS9YJzc3pZn/52sRLL6XnX++5J3znO2l48OB6R2pmVl1NneNZP8W+2ej/i4i/lBpVQY3c\nVLV4ccvaxAMPwGabrWxy2nPPdObTWmvVO1Iz603K7uMYDSyIiPey8XWAoRHxbGd2WIZGSRzLl6fb\nc1SSxD33pMee7r77yg7s8eNhww3rHamZ9XalP3Mc2DMilmbj/YC7I2KPzuywDPVKHG+8AffdtzJJ\n3HdfSgr52sROO0Gfrrxc0sysC5TdOd6nkjQAImJpljx6lQh48smWt+t45pl0V9g994SvfAWuuAI2\n2aTekZqZlauWxPGypIMi4iYASQcDi8oNq/7eemvVm/8NGLCyNnHiibDLLtC3b70jNTPrXrU0VW1F\nOrNqc0CkmxYeGxFPlR9ebVa3qSoC5sxpWZuYPTslhvyV2Jtv3oVBm5nVUbc8yEnSAICIeEvS0IhY\n2JkdlqFo4qg85jR/JXafPiuTxJ57pluJr+2rVcxsDdVdiWMwcAhwFLB9RDTM8Xd7iSMC5s9vWZuY\nOTM95jRfm/CDicysNyktcWSn3h5MSha7AgOBzwB3RsTyzuywDPnEsWQJTJ3asjbR3NzyTKcPftCP\nOTWz3q2UxCHpamA88Bfgd8DfgKciYnRnAy2LpPjmN4N774UZM2DbbVvWJkaPdm3CzCyvrNNxdwBe\nAmYBsyKiWVL9r7Jrw4YbwjnnwB57+FbiZmZl6qipajvgSODzwMvAdsBOjdQxDo1z5biZWU/RXZ3j\nHyQlkcOB5yJiz87ssAxOHGZmxXRL4sjtTMDeEXFnZ3ZYBicOM7NiujVxNCInDjOzYlYncfjRQGZm\nVogTh5mZFdLhTQ4lrU26YnxUfvmI+EF5YZmZWaOq5e64NwKvAw8BS8oNx8zMGl0tiWNYROxXeiRm\nZtYj1NLHcY+kD5QeiZmZ9Qi1PI/jMWAMMIfUVCUgImLn8sOrjU/HNTMrpuxHx+7fmQ2bmdmaqaYL\nACXtAuydjd4VETNKjaog1zjMzIop9QJASaeSHh27SfZ3paSvd2ZnZmbW89XSx/EwMCEi3s7G1wPu\ndR+HmVnPVfYtRwQ058abs2lmZtYL1dI5fhlwn6QbsvHPAJeUF5KZmTWyWjvHdwP2ykbviohppUZV\nkJuqzMyKKeuZ4wMj4g1JG1SbHxGvdmaHZXDiMDMrpqzE8T8R8Q+S5gD5hSoXAG7ZmR2WwYnDzKwY\nP8jJicPMrJCyr+O4rZZpZmbWO7R5VpWk/sC6wEaShrDyFNyBwBbdEJuZmTWg9mocXyY9g2O77H/l\n70bgl7XuQNJ+kh6X9ISkb1eZP1jSHyTNkDRF0g6t5jdJmirpplr3aWZm5anlyvGvR8R/dGrjUhPw\nBPBx4AXzZr1jAAANCUlEQVTgAeCIiHg8t8z5wJsRcbakbYELImLf3Px/Aj4IDIyIg9rYj/s4zMwK\nKLWPIyL+Q9JOkg6XdGzlr8btjwOejIi5EbEMuBY4uNUyOwC3Z/uaDYyStDGApGHAAcDFNe7PzMxK\nVkvn+FnAf2R/+wDnA1WP/KvYApifG3+OVftHZgCfy/Y1DhgBDMvm/RT4Z1qeDmxmZnVUy72qDiU1\nNb0YEccDuwCDujCGc4EhkqYCXwOmAc2SDgQWRsR0Use8749lZtYAarlX1bsRsVzS+5IGAi8Bw2vc\n/vOkGkTFsGzaChHxJvClyrikZ4BngCOAgyQdAKwDrC/pioio2kw2adKkFcMTJ05k4sSJNYZoZrbm\nmzx5MpMnT+6SbdXSOX4h8F3SD/lpwFvA9Kz20dG6awGzSTWWBcD9wJERMSu3zCDgnYhYJulE4MMR\ncVyr7XwUOM2d42ZmXaPUR8dGxFezwf+S9GfS2U0P17LxiGiWdDJwK6lZ7JKImCXpy2l2XARsD1wu\naTkwEzihMy/EzMy6R3v3qtqtvRUjYmopEXWCaxxmZsWUdZPDO7LB/sDupLOfBOwMPBgREzqzwzI4\ncZiZFVPKdRwRsU9E7EPqm9gtInaPiA8CY2nVwW1mZr1HLafjbhsRj1RGIuJRUr+EmZn1QrWcjvuw\npIuBK7PxLwA1dY6bmdmap5bTcfsDXwE+kk26E/jPiHiv5Nhq5j4OM7Ni/CAnJw4zs0JKuY5D0nUR\ncbikR6hyr6iI2LkzOzQzs56tvdNxN4uIBZJGVpsfEXNLjawA1zjMzIpxU5UTh5lZIWU1Vb1J9duZ\ni3S7kIGd2aGZmfVsbSaOiFi/OwMxM7OeoZbrOACQtAnp9iMARMS8UiIyM7OGVssTAA+S9CQwB/gb\n8CxwS8lxmZlZg6rlliNnA+OBJyJiNOnZGlNKjcrMzBpWLYljWUS8AjRJaoqIO0h3yzUzs16olj6O\n1yQNIN1q5CpJLwFvlxuWmZk1qlruVbUe8B7pNNwvAIOAq7JaSEPwdRxmZsWU9SCnC4CrI+Lu1Qmu\nOzhxmJkVU8qDnIAngB9JelbS+ZLGdi48MzNbk9TSVDUSOCL7Wwe4BrgmIp4oP7zauMZhZlZMt92r\nKqt1XArsHBFrdWaHZXDiMDMrpqymqsrG+0j6tKSrSBf+zQY+15mdmZlZz9de5/gngCOBA4D7gWuB\nGyOi4U7FdY3DzKyYss6quh24Gvh9RCxejfhK58RhZlaMn8fhxGFmVkipfRxmZmZ5ThxmZlaIE4eZ\nmRXixGFmZoU4cZiZWSFOHGZmVogTh5mZFeLEYWZmhThxmJlZIU4cZmZWiBOHmZkVUnrikLSfpMcl\nPSHp21XmD5b0B0kzJE2RtEM2fZik2yXNlPSIpFPKjtXMzDpW6k0OJTWRHkH7ceAF4AHgiIh4PLfM\n+cCbEXG2pG2BCyJiX0mbAptGxHRJA4CHgIPz6+a24ZscmpkV0Mg3ORwHPBkRcyNiGemZHge3WmYH\n4HaAiJgNjJK0cUS8GBHTs+lvAbOALUqO18zMOlB24tgCmJ8bf45Vf/xnkD1RUNI4YAQwLL+ApFHA\nrsB9JcVpZmY16lPvAIBzgZ9Lmgo8AkwDmiszs2aq64FTs5pHVZMmTVoxPHHiRCZOnFhSuGZmPc/k\nyZOZPHlyl2yr7D6O8cCkiNgvGz8DiIg4r5115gAfiIi3JPUB/ge4JSJ+3s467uMwMyugkfs4HgDG\nSBopqR9wBHBTfgFJgyT1zYZPBP6Wq1lcCjzWXtIwM7PuVWpTVUQ0SzoZuJWUpC6JiFmSvpxmx0XA\n9sDlkpYDM4ETACR9GPgC8IikaUAA342IP5cZs5mZtc/PHDcz64UauanKzMzWME4cZmZWiBOHmZkV\n4sRhZmaFOHGYmVkhThxmZlaIE4eZmRXixGFmZoU4cZiZWSFOHGZmVogTh5mZFeLEYWZmhThxmJlZ\nIU4cZmZWiBOHmZkV4sRhZmaFOHGYmVkhThxmZlaIE4eZmRXixGFmZoU4cZiZWSFOHGZmVogTh5mZ\nFeLEYWZmhThxmJlZIU4cZmZWiBOHmZkV4sRhZmaFOHGYmVkhThxmZlaIE4eZmRXixGFmZoU4cZiZ\nWSFOHGZmVogTh5mZFVJ64pC0n6THJT0h6dtV5g+W9AdJMyRNkbRDreuamVn3KzVxSGoCfgl8CtgR\nOFLSdq0W+y4wLSJ2Ab4I/KLAuj3G5MmT6x1CTRxn13KcXctxNoayaxzjgCcjYm5ELAOuBQ5utcwO\nwO0AETEbGCVp4xrX7TF6ygfJcXYtx9m1HGdjKDtxbAHMz40/l03LmwF8DkDSOGAEMKzGdc3MrJs1\nQuf4ucAQSVOBrwHTgOb6hmRmZm1RRJS3cWk8MCki9svGzwAiIs5rZ505wAeAnWpdV1J5L8LMbA0V\nEerMen26OpBWHgDGSBoJLACOAI7MLyBpEPBORCyTdCLwt4h4S1KH61Z09sWbmVlxpSaOiGiWdDJw\nK6lZ7JKImCXpy2l2XARsD1wuaTkwEzihvXXLjNfMzDpWalOVmZmteRqhc7wmtVwMKOkXkp6UNF3S\nrt0dYxZDRxc8flTSa5KmZn9n1iHGSyQtlPRwO8s0Qlm2G2cjlGUWxzBJt0uaKekRSae0sVxdy7SW\nOOtdppLWlnSfpGlZnOe0sVy9y7LDOOtdlq1iacpiuKmN+cXKMyIa/o+U4J4CRgJ9genAdq2W2R/4\n32z4Q8CUBo3zo8BNdS7PvYBdgYfbmF/3sqwxzrqXZRbHpsCu2fAAYHaDfj5ribPuZQqsm/1fC5gC\nfLjRyrLGOOtelrlY/gm4slo8nSnPnlLjqOViwIOBKwAi4j5gkKSh3RtmzRct1rUzPyL+DixuZ5FG\nKMta4oQ6lyVARLwYEdOz4beAWax6zVHdy7TGOKH+n893ssG1SQdjrT8DdS/LbN8dxQkN8PmUNAw4\nALi4jUUKl2dPSRy1XAzYepnnqyxTtlovWpyQVQn/N39vrgbSCGVZq4YqS0mjSLWk+1rNaqgybSdO\nqHOZZs0q04AXgckR8VirRRqiLGuIExrj8/lT4J+Btjq0C5dnT0kca5KHgBERsSvpXlx/rHM8PVlD\nlaWkAcD1wKnZEX1D6iDOupdpRCyPiLGkO0h8RNJHuzuGWtQQZ93LUtKBwMKspim6qAbUUxLH86Rb\nkVQMy6a1XmZ4B8uUrcM4I+KtShU3Im4B+kraoPtCrEkjlGWHGqksJfUh/Rj/JiJurLJIQ5RpR3E2\nUplGxBvA/wK7t5rVEGVZ0VacDVKWHwYOkvQMcA2wj6QrWi1TuDx7SuJYcTGgpH6kiwFbnx1wE3As\nrLhi/bWIWNi9YXYcZ77tUOneXIqIV7s3zLR72j76aISyrGgzzgYqS4BLgcci4udtzG+UMm03znqX\nqaSNlC4KRtI6wCdIJ5nk1b0sa4mz3mUJEBHfjYgREbEl6ffo9og4ttVihcuz7CvHu0TUcCFhRNws\n6QBJTwFvA8c3YpzAoZK+AiwD3gU+391xSroamAhsKGkecBbQjwYqy1ripAHKMovzw8AXgEeyNu8g\nPS5gJA1UprXESf3LdDPSBcEifYd+ExG3Ndp3vZY4qX9Ztml1y9MXAJqZWSE9panKzMwahBOHmZkV\n4sRhZmaFOHGYmVkhThxmZlaIE4eZmRXixGG9nqQ3s/8jJVV9yuRqbPs7rcb/3pXbN6sHJw6zlTd/\nGw0cVWRFSWt1sMh3W+woYq8i2zdrRE4cZiv9ENgre+DNqdndT8/PHtgzXdKJsOIBPXdKupH0uGMk\n3SDpAaUHJP1jNu2HwDrZ9n6TTXuzsjNJ/54tP0PS4blt3yHpd5JmVdbL5p0r6dEslvO7rVTMWukR\ntxwx6yZnAKdFxEEAWaJ4LSI+lN177G5Jt2bLjgV2jIh52fjxEfGapP7AA5J+HxHfkfS1iNgtt4/I\ntn0IsHNEfEDSJtk6f8uW2RXYgXS77rsl7Qk8DnwmIrbL1h9YViGYdcQ1DrO2fRI4Nruv033ABsDW\n2bz7c0kD4BuSppOeBDcst1xbPky6WykR8RIwGdgjt+0Fke4HNB0YBbwOvCvpYkmfJd37yKwunDjM\n2ibg6xExNvvbKiL+ms17e8VC6TkMHwM+lD17YTrQP7eNWvdVsSQ33Az0iYhm0hMmrwf+Afhz4Vdj\n1kWcOMxW/mi/Cayfm/4X4KvZMyyQtLWkdausPwhYHBFLJG0HjM/NW1pZv9W+7gI+n/WjbAzsDdzf\nZoBpv4Mj4s/AN4Gda395Zl3LfRxmK8+qehhYnjVN/Toifq70iNWp2e2zXwI+U2X9PwP/R9JMYDZw\nb27eRcDDkh6KiGMq+4qIG7JnH8wAlgP/HBEvSdq+jdgGAjdmfSgA/9T5l2u2enxbdTMzK8RNVWZm\nVogTh5mZFeLEYWZmhThxmJlZIU4cZmZWiBOHmZkV4sRhZmaFOHGYmVkh/x/N8i/t7044/QAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116709050>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_data(validation_accuracy, 'accuracy', [0, 4, 0.9, 1],\n",
    "          'Iterations', 'Validation Accuracy', \n",
    "          'Validation accuracy over iterations Single layer NN')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model accuracy:  0.9709\n"
     ]
    }
   ],
   "source": [
    "test_pred = []\n",
    "for i in range(len(test_dataset)):\n",
    "    test_inp_line = test_dataset[i]\n",
    "    hidden_inp = np.dot(hidden_wts, test_inp_line)\n",
    "    hidden_out = 1/(1 + np.exp(-1 * hidden_inp))\n",
    "    second_inp = np.dot(neural_out_wts,hidden_out)\n",
    "    neural_out_line = 1/(1 + np.exp(-1 * second_inp))\n",
    "    test_pred.append(neural_out_line)\n",
    "print \"Model accuracy: \",accuracy(\n",
    "    raw_test_labels,one_hot_encoding(np.array(test_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the improvements of classification on the test set are massive after using a single layer neural network. We got an accuracy of __97.1%__ on the test dataset reserved from the MNIST dataset.\n",
    "\n",
    "Before we start with the __Convolutional Neural Network__, we are going to preprocess the USPS Dataset. \n",
    "After preprocessing the dataset, to have similar format as the MNIST dataset (since we have trained our models on this dataset), we are going to try and predict the accuracy of the __Logistic Regression__ model and the __Single Layer Neural Network__ model on the __USPS Datset__.\n",
    "\n",
    "This dataset has been completely hidden/unknown for our trained models, and having studied the __No Free Lunch Theorem__, we can expect a massive drop in the prediction accuracies.\n",
    "\n",
    "Before moving on to the USPS Dataset though, I would like to comment on the __'No Free Lunch Theorem'__\n",
    "\n",
    "## No Free Lunch (NFL) Theorem\n",
    "\n",
    "A model is a simplified representation of reality, and the simplifications are made to discard unnecessary detail and allow us to focus on the aspect of reality that we want to understand.  These simplifications are grounded on assumptions; these assumptions may hold in some situations, but may not hold in other situations.  This implies that a model that explains a certain situation well may fail in another situation.\n",
    "\n",
    "The __“No Free Lunch”__ theorem states that there is __no one model that works best for every problem.__  \n",
    "\n",
    "The assumptions of a great model for one problem may not hold for another problem, so it is common in machine learning to try multiple models and find one that works best for a particular problem.  \n",
    "\n",
    "This is __especially true in supervised learning__ problem, which we are trying to solve in this project.\n",
    "\n",
    "__Validation__ or __Cross-Validation__ is commonly used to assess the predictive accuracies of multiple models of varying complexity to find the best model.\n",
    "\n",
    "With an understanding of the theorem, let us move to the next section, in which we shall be able to verify it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## USPS Data \n",
    "\n",
    "### Data Processing\n",
    "\n",
    "So the USPS Dataset comprises of about 20000 images, which have a pixel resolution of 100ppi. But as we have trained the models on the MNIST dataset, the models expect a 784-dimensional feature vector as an input.\n",
    "\n",
    "Moreover, the pixels are 8-bit encoded, ie. have values between 0 and 255. Our model was trained to have feature values between 0 and 1.\n",
    "\n",
    "Because of this, we need to preprocess the USPS data images. We shall __resize__ and __rescale__ these images to take care of the above mentioned problems.\n",
    "\n",
    "The next sections of code define functions necessary for resizing and scaling. \n",
    "    \n",
    "    1. Then we iterate through each image and read it into a matrix as a \n",
    "    grayscale image.\n",
    "    \n",
    "    2. Resize and scale the read image\n",
    "    \n",
    "    3. Split it into input feature and an output label.\n",
    "    \n",
    "    4. Flatten the image matrix into a 784 dimension vector.\n",
    "    \n",
    "    5. One-hot encode the output labels to have a 10 dimensional output vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_to_data = \"./USPSdata/Numerals/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def resize_and_scale(img, size, scale):\n",
    "    img = cv2.resize(img, size)\n",
    "    return 1 - np.array(img, \"float32\")/scale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "img_list = os.listdir(path_to_data)\n",
    "sz = (28,28)\n",
    "validation_usps = []\n",
    "validation_usps_label = []\n",
    "for i in range(10):\n",
    "    label_data = path_to_data + str(i) + '/'\n",
    "    img_list = os.listdir(label_data)\n",
    "    for name in img_list:\n",
    "        if '.png' in name:\n",
    "            img = cv2.imread(label_data+name)\n",
    "            img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "            resized_img = resize_and_scale(img, sz, 255)\n",
    "            validation_usps.append(resized_img.flatten())\n",
    "            validation_usps_label.append(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(19999, 784)\n",
      "(19999, 10)\n"
     ]
    }
   ],
   "source": [
    "validation_usps = np.array(validation_usps)\n",
    "print validation_usps.shape\n",
    "validation_usps_label= np.array(validation_usps_label)\n",
    "print reformat(validation_usps_label).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As verified above, we have input matrix as a 19999 x 784 matrix, and an output matrix of dimensions 19999 x 10.\n",
    "\n",
    "Now, that we have preprocessed the image as required by our trained models, we can test the performance of our models on this dataset.\n",
    "\n",
    "### Logistic Regression on USPS Data\n",
    "\n",
    "In this section, we are going to apply the pre-trained __Logistic Regression(LR)__ classifier on the USPS dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pred_output_usps_lr = np.dot(add_ones(validation_usps), weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3629181459072954"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(validation_usps_label, one_hot_encoding(pred_output_usps_lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__The accuracy of LR Model on USPS Dataset is 36.3%__\n",
    "\n",
    "As we can verify, the LR model performs very poorly on the USPS Dataset. This is a good verification of the No free Lunch theorem. As this dataset is completely unknown to our Logistic classifier, the model performs poorly.\n",
    "\n",
    "Now let us explore the results of Single layer Neural Network model on this dataset.\n",
    "\n",
    "### Single Layer Neural Network on USPS Data\n",
    "\n",
    "In this section we shall apply the Neural Network classifier on the USPS Dataset. \n",
    "\n",
    "We hypothesize a poor performance of the model, in keeping with the No Free Lunch Theorem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model accuracy:  0.492274613731\n"
     ]
    }
   ],
   "source": [
    "test_pred = []\n",
    "for i in range(len(validation_usps)):\n",
    "    test_inp_line = validation_usps[i]\n",
    "    hidden_inp = np.dot(hidden_wts, test_inp_line)\n",
    "    hidden_out = 1/(1 + np.exp(-1 * hidden_inp))\n",
    "    second_inp = np.dot(neural_out_wts,hidden_out)\n",
    "    neural_out_line = 1/(1 + np.exp(-1 * second_inp))\n",
    "    test_pred.append(neural_out_line)\n",
    "print \"Model accuracy: \",accuracy(\n",
    "    validation_usps_label, one_hot_encoding(np.array(test_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results of Single Layer Neural Network on USPS Data is as follows:\n",
    "\n",
    "#### SNN on USPS Accuracy : 49.23 %\n",
    "\n",
    "As we can see, the model performs somewhat better than the Logistic classifier, as was the case for even MNIST Dataset.\n",
    "\n",
    "But due to lack of training the model on this data, and by NFL, the model performance is poor.\n",
    "\n",
    "Now let us implement a convolutional neural network in the following section.\n",
    "\n",
    "## Convolutional Neural Network\n",
    "\n",
    "These models are the successors of the single layer Neural Network we implemented previously, and belong to the class of __Deep Learning Networks__. They have derived their name, from the fact that, they stack many layers of perceptrons one after the other, to form a deep representation.\n",
    "\n",
    "Each layer has a different task and an associated activation function. The main parts of a Convolutional Neural Network or __ConvNets__ as they are referred to, are as follows:\n",
    "    \n",
    "    1. Input Layer\n",
    "    \n",
    "    2. Convolutional Layer\n",
    "    \n",
    "    3. ReLu Layer\n",
    "    \n",
    "    4. Max Pooling Layer\n",
    "    \n",
    "    5. Dropout Layer(Only for training)\n",
    "    \n",
    "    6. Fully Connected Layers\n",
    "    \n",
    "__Convolutional Layers__ and __ReLu layers__ are responsible for feature extractions in the given image. They start from representing simple patterns in the images(such as edges, corners) and represent more complex patterns in the deeper layers, which are usually representations of combined simple layers.\n",
    "\n",
    "__Max Pooling Layers__ are used for reducing dimensions(__strides__) by eliminating redundant data in the patterns, and storing the important features from the convolutional layers.\n",
    "\n",
    "__Dropout Layers__ are used for training the network to generalise better on data. The layer removes random connections in between perceptrons of different layers, which makes the system more robust on noisy data, and different representations of similar images.\n",
    "\n",
    "__Fully Connected Layers__ are at the end of the network, which train based on the extracted features of previous layers mentioned above. They are similar to The previously trained neural network, only with more hidden layers.\n",
    "\n",
    "With this explanation of ConvNets, we are well equipped to train a CNN on our MNIST Dataset. We shall be using __Tensorflow__ for building a CNN, and train on our CPU.\n",
    "\n",
    "\n",
    "As always, let us first conver the MNIST data, into tensors, so that we can feed it to the tensorflow graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "training_inp = np.reshape(train_dataset, (50000, 28, 28))\n",
    "valid_inp = np.reshape(valid_dataset, (10000, 28, 28))\n",
    "test_inp = np.reshape(test_dataset, (10000, 28, 28))\n",
    "usps_data = np.reshape(validation_usps, (19999, 28, 28))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Training set', (50000, 28, 28, 1), (50000, 10))\n",
      "('Validation set', (10000, 28, 28, 1), (10000, 10))\n",
      "('Test set', (10000, 28, 28, 1), (10000, 10))\n",
      "('USPS set', (19999, 28, 28, 1), (19999, 10))\n"
     ]
    }
   ],
   "source": [
    "image_size = 28\n",
    "num_labels = 10\n",
    "num_channels = 1 # grayscale\n",
    "\n",
    "def reformat(dataset, labels):\n",
    "  dataset = dataset.reshape(\n",
    "    (-1, image_size, image_size, num_channels)).astype(np.float32)\n",
    "  labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)\n",
    "  return dataset, labels\n",
    "train_dataset, train_labels = reformat(training_inp, raw_train_labels)\n",
    "valid_dataset, valid_labels = reformat(valid_inp, raw_valid_labels)\n",
    "test_dataset, test_labels = reformat(test_inp, raw_test_labels)\n",
    "usps_dataset, usps_labels = reformat(usps_data, validation_usps_label)\n",
    "print('Training set', train_dataset.shape, train_labels.shape)\n",
    "print('Validation set', valid_dataset.shape, valid_labels.shape)\n",
    "print('Test set', test_dataset.shape, test_labels.shape)\n",
    "print('USPS set', usps_dataset.shape, usps_labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As printed out in previous section, we have tensors representing our __MNIST Data__ and the __USPS Data__.\n",
    "\n",
    "#### Note: USPS data and MNIST Test set are both not used in training phase. They shall only be used to output the final accuracy of our trained CNN.\n",
    "\n",
    "We are just building the graph in a single process, because CNNs are extremely memory intensive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def accuracy(predictions, labels):\n",
    "    return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))\n",
    "          / predictions.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convolutional Neural Network Parameters\n",
    "\n",
    "In our model, we shall se having the following parameters:\n",
    "    1. Batch Size: 16\n",
    "    \n",
    "    2. Convolutional Patch Size: 5 x 5\n",
    "    \n",
    "    3. Channels: 1 (Grayscale Image)\n",
    "    \n",
    "    4. Regularisation beta: 0.001\n",
    "    \n",
    "    5. Dropout Probability: 0.5\n",
    "    \n",
    "    6. Fully Connected Layer: Hidden layer size - 64\n",
    "    \n",
    "    7. Output Layer: 10 neurons\n",
    "    \n",
    "    8. Stride length: 2 pixels\n",
    "    \n",
    "Having defined the convolutional network parameters, lets train the CNN in following section. \n",
    "\n",
    "We shall get the accuracy of our model on MNIST test set and USPS Data at the end of training phase, and after predicting on the above mentioned sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "batch_size = 16\n",
    "patch_size = 5\n",
    "depth = 16\n",
    "num_hidden = 64\n",
    "beta_regul = 1e-3\n",
    "drop_out = 0.5\n",
    "\n",
    "graph = tf.Graph()\n",
    "\n",
    "with graph.as_default():\n",
    "\n",
    "  # Input data.\n",
    "  tf_train_dataset = tf.placeholder(\n",
    "    tf.float32, shape=(batch_size, image_size, image_size, num_channels))\n",
    "  tf_train_labels = tf.placeholder(tf.float32, \n",
    "                                   shape=(batch_size, num_labels))\n",
    "  tf_valid_dataset = tf.constant(valid_dataset)\n",
    "  tf_test_dataset = tf.constant(test_dataset)\n",
    "  tf_usps_dataset = tf.constant(usps_dataset)\n",
    "  beta_regul = tf.placeholder(tf.float32)\n",
    "  global_step = tf.Variable(0)\n",
    "  \n",
    "  # Variables.\n",
    "  layer1_weights = tf.Variable(tf.truncated_normal(\n",
    "      [patch_size, patch_size, num_channels, depth], stddev=0.1))\n",
    "  layer1_biases = tf.Variable(tf.zeros([depth]))\n",
    "  layer2_weights = tf.Variable(tf.truncated_normal(\n",
    "      [patch_size, patch_size, depth, depth], stddev=0.1))\n",
    "  layer2_biases = tf.Variable(tf.constant(1.0, shape=[depth]))\n",
    "  size3 = ((image_size - patch_size + 1) // 2 - patch_size + 1) // 2\n",
    "  layer3_weights = tf.Variable(tf.truncated_normal(\n",
    "      [size3 * size3 * depth, num_hidden], stddev=0.1))\n",
    "  layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))\n",
    "  layer4_weights = tf.Variable(tf.truncated_normal(\n",
    "      [num_hidden, num_hidden], stddev=0.1))\n",
    "  layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))\n",
    "  layer5_weights = tf.Variable(tf.truncated_normal(\n",
    "      [num_hidden, num_labels], stddev=0.1))\n",
    "  layer5_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))\n",
    "  \n",
    "  # Model.\n",
    "  def model(data, keep_prob):\n",
    "    # C1 input 28 x 28\n",
    "    conv1 = tf.nn.conv2d(data, layer1_weights, \n",
    "                         [1, 1, 1, 1], padding='VALID')\n",
    "    bias1 = tf.nn.relu(conv1 + layer1_biases)\n",
    "    # S2 input 24 x 24\n",
    "    pool2 = tf.nn.max_pool(bias1, [1, 2, 2, 1], \n",
    "                           [1, 2, 2, 1], padding='VALID')\n",
    "    # C3 input 12 x 12\n",
    "    conv3 = tf.nn.conv2d(pool2, layer2_weights, \n",
    "                         [1, 1, 1, 1], padding='VALID')\n",
    "    bias3 = tf.nn.relu(conv3 + layer2_biases)\n",
    "    # S4 input 8 x 8\n",
    "    pool4 = tf.nn.max_pool(bias3, [1, 2, 2, 1], \n",
    "                           [1, 2, 2, 1], padding='VALID')\n",
    "    # F5 input 4 x 4\n",
    "    shape = pool4.get_shape().as_list()\n",
    "    reshape = tf.reshape(pool4, \n",
    "                         [shape[0], shape[1] * shape[2] * shape[3]])\n",
    "    hidden5 = tf.nn.relu(tf.matmul(\n",
    "            reshape, layer3_weights) + layer3_biases)\n",
    "    # F6\n",
    "    drop5 = tf.nn.dropout(hidden5, keep_prob)\n",
    "    hidden6 = tf.nn.relu(tf.matmul(\n",
    "            hidden5, layer4_weights) + layer4_biases)\n",
    "    drop6 = tf.nn.dropout(hidden6, keep_prob)\n",
    "    return tf.matmul(drop6, layer5_weights) + layer5_biases\n",
    "  \n",
    "  # Training computation.\n",
    "  logits = model(tf_train_dataset, drop_out)\n",
    "  loss = tf.reduce_mean(\n",
    "    tf.nn.softmax_cross_entropy_with_logits(\n",
    "        logits, tf_train_labels)) \n",
    "        + beta_regul * (tf.nn.l2_loss(layer1_weights) \n",
    "                        + tf.nn.l2_loss(layer2_weights) \n",
    "                        + tf.nn.l2_loss(layer3_weights) \n",
    "                        + tf.nn.l2_loss(layer4_weights)\n",
    "                        + tf.nn.l2_loss(layer5_weights))\n",
    "    \n",
    "  # Optimizer.\n",
    "  learning_rate = tf.train.exponential_decay(\n",
    "        0.05, global_step, 1000, 0.85, staircase=True)\n",
    "  optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "                        .minimize(loss, global_step=global_step)\n",
    "  \n",
    "  # Predictions for the training, validation, and test data.\n",
    "  train_prediction = tf.nn.softmax(logits)\n",
    "  valid_prediction = tf.nn.softmax(model(tf_valid_dataset, 1.0))\n",
    "  test_prediction = tf.nn.softmax(model(tf_test_dataset, 1.0))\n",
    "  usps_prediction = tf.nn.softmax(model(tf_usps_dataset, 1.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initialized\n",
      "Minibatch loss at step 0: 3.529994\n",
      "Minibatch accuracy: 6.2%\n",
      "Validation accuracy: 10.6%\n",
      "Minibatch loss at step 1000: 0.263206\n",
      "Minibatch accuracy: 93.8%\n",
      "Validation accuracy: 94.6%\n",
      "Minibatch loss at step 2000: 0.237529\n",
      "Minibatch accuracy: 93.8%\n",
      "Validation accuracy: 96.4%\n",
      "Minibatch loss at step 3000: 0.241817\n",
      "Minibatch accuracy: 93.8%\n",
      "Validation accuracy: 96.6%\n",
      "Minibatch loss at step 4000: 0.130609\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 97.8%\n",
      "Minibatch loss at step 5000: 0.148814\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 97.6%\n",
      "Minibatch loss at step 6000: 0.218663\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 97.9%\n",
      "Minibatch loss at step 7000: 0.214761\n",
      "Minibatch accuracy: 93.8%\n",
      "Validation accuracy: 98.3%\n",
      "Minibatch loss at step 8000: 0.112256\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.2%\n",
      "Minibatch loss at step 9000: 0.119111\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.1%\n",
      "Minibatch loss at step 10000: 0.292125\n",
      "Minibatch accuracy: 93.8%\n",
      "Validation accuracy: 98.4%\n",
      "Minibatch loss at step 11000: 0.178908\n",
      "Minibatch accuracy: 93.8%\n",
      "Validation accuracy: 98.3%\n",
      "Minibatch loss at step 12000: 0.114231\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.5%\n",
      "Minibatch loss at step 13000: 0.105859\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.5%\n",
      "Minibatch loss at step 14000: 0.160570\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.5%\n",
      "Minibatch loss at step 15000: 0.112208\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.7%\n",
      "Minibatch loss at step 16000: 0.112096\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.5%\n",
      "Minibatch loss at step 17000: 0.108502\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.5%\n",
      "Minibatch loss at step 18000: 0.107968\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.5%\n",
      "Minibatch loss at step 19000: 0.120067\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.5%\n",
      "Minibatch loss at step 20000: 0.140188\n",
      "Minibatch accuracy: 100.0%\n",
      "Validation accuracy: 98.5%\n",
      "Test accuracy: 98.7%\n",
      "USPS data accuracy: 63.0%\n"
     ]
    }
   ],
   "source": [
    "num_steps = 20001\n",
    "\n",
    "with tf.Session(graph=graph) as session:\n",
    "  tf.initialize_all_variables().run()\n",
    "  print('Initialized')\n",
    "  for step in range(num_steps):\n",
    "    offset = (step * batch_size) % \n",
    "            (train_labels.shape[0] - batch_size)\n",
    "    batch_data = train_dataset[offset:(offset + batch_size), :, :, :]\n",
    "    batch_labels = train_labels[offset:(offset + batch_size), :]\n",
    "    feed_dict = {tf_train_dataset : batch_data, \n",
    "                 tf_train_labels : batch_labels, beta_regul : 1e-3}\n",
    "    _, l, predictions = session.run(\n",
    "      [optimizer, loss, train_prediction], feed_dict=feed_dict)\n",
    "    if (step % 1000 == 0):\n",
    "      print('Minibatch loss at step %d: %f' % (step, l))\n",
    "      print('Minibatch accuracy: %.1f%%' % \n",
    "            accuracy(predictions, batch_labels))\n",
    "      print('Validation accuracy: %.1f%%' % \n",
    "            accuracy(valid_prediction.eval(), valid_labels))\n",
    "  print('Test accuracy: %.1f%%' % accuracy(\n",
    "        test_prediction.eval(), test_labels))\n",
    "  print('USPS data accuracy: %.1f%%' % accuracy(\n",
    "            usps_prediction.eval(), usps_labels))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We have trained the data for 20000 epochs, after which we have stopped in order to avoid overfitting.\n",
    "\n",
    "Though we have printed out the results above, let us formally state them below:\n",
    "\n",
    "#### CNN accuracy MNIST test: 98.7 %\n",
    "\n",
    "#### CNN accuracy USPS data: 63.0 %\n",
    "\n",
    "There is a massive improvement on the USPS Data, as compared to the MNIST Dataset. This proves that __CNNs are able to better generalise on unseen data__ than single layer neural networks or logistic classifier model.\n",
    "\n",
    "Again, we face the consequences of the __No free Lunch Theorem__, and hence performance drops even for CNNs on USPS Data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this project we were able to achieve the following:\n",
    "\n",
    "    1. Train and implement 3 machine learning models on MNIST Dataset, namely:\n",
    "    \n",
    "        i. Logistic Regression Classifier\n",
    "        \n",
    "        ii. Single Layer Neural Network\n",
    "        \n",
    "        iii. Convolutional Neural Network\n",
    "    \n",
    "    2. The model performances are as follows, on test MNIST and USPS Data:\n",
    "        \n",
    "        i. MNIST Test set:\n",
    "            \n",
    "            a. Logistic Regression Classifier: 92.3%\n",
    "            \n",
    "            b. Single Layer Neural Network:    97.1%\n",
    "            \n",
    "            c.Convolutional Neural Network:    98.7%\n",
    "        \n",
    "        ii. USPS Data set:\n",
    "            \n",
    "            a. Logistic Regression Classifier: 36.3%\n",
    "            \n",
    "            b. Single Layer Neural Network:    49.2%\n",
    "            \n",
    "            c.Convolutional Neural Network:    63.0%\n",
    "    \n",
    "    3. We learnt about the 'No Free Lunch Theorem' in Machine Learning,\n",
    "    and also verified its truth by testing the performance of our model \n",
    "    on the USPS Dataset.\n",
    "    \n",
    "    4. In conclusion, Convolutional Neural networks seem the way forward, \n",
    "    and we have verified the impact of Deep Learning on simple task of handwriting \n",
    "    recognition. CNNs are performing some state of the art work on various \n",
    "    Computer Vision tasks as well, in various fields such as medicine, \n",
    "    automated cars, object detection, etc."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
